Course contentsShow
← System Design
Lesson 767 of 1,91920. Data Pipelines and ETLFree lesson

Airflow Operators and Executors

Understand operators for diverse tasks and executors (local, Celery, Kubernetes) for scaling Airflow.

Airflow Operators and Executors

What You'll Learn

You'll discover how Airflow uses operators to define individual tasks and executors to determine where and how those tasks actually run.

Operators: The Building Blocks of Tasks

In Airflow, an operator is a template for a specific type of work. Think of operators as pre-built tools in a toolbox—each designed for a particular job:

  • BashOperator: Runs shell commands
  • PythonOperator: Executes Python functions
  • SqlOperator: Runs SQL queries
  • EmailOperator: Sends email notifications
  • SensorOperator: Waits for conditions (like file arrival)

You don't write the low-level execution logic; you just configure the operator with parameters. This makes DAGs readable and maintainable—each task clearly states what it does.

Executors: Where Tasks Actually Run

While operators define what to do, executors determine where and how tasks execute. They're Airflow's execution engines:

LocalExecutor

Runs tasks as separate processes on the same machine where Airflow's scheduler runs. Good for development or small workloads, but doesn't scale beyond one server.

CeleryExecutor

Distributes tasks across multiple worker machines using Celery (a distributed task queue). Workers pull tasks from a message broker like Redis or RabbitMQ. This scales horizontally—add more workers as your workload grows.

KubernetesExecutor

Launches each task as a separate Kubernetes pod. Tasks get isolated compute resources and spin down after completion. Ideal for cloud-native environments with variable workloads—you only pay for what you use.

Think of it like delivery methods: LocalExecutor is doing everything yourself, CeleryExecutor is a team of delivery drivers waiting for packages, and KubernetesExecutor is ordering on-demand drivers per delivery.

Key Takeaway: Operators provide reusable task templates for diverse data operations, while executors (Local, Celery, Kubernetes) control how Airflow scales from single-machine to distributed execution.