Configuring Airflow Scheduler and Executor Architecture

Understand the interaction between the Airflow scheduler, metadata database, and executors to manage the lifecycle and distribution of your pipeline tasks.

In this lesson, you will learn how the Airflow scheduler and executor work in tandem to transform your DAG definitions into actual, running data tasks.

At the heart of Airflow’s architecture are three primary components: the Scheduler, the Metadata Database, and the Executor. Think of them as the brain, the memory, and the muscle of your orchestration layer.

The Lifecycle of a Task

The Scheduler (The Brain): The scheduler is a persistent service that monitors your DAG files. It parses your Python files, identifies which tasks are ready to run (based on dependencies and schedules), and updates the metadata database to set their state to queued.
The Metadata Database (The Memory): Airflow stores the state of every task instance here. When the scheduler sees a task is ready, it records the queued status. The executor then queries this database to see which tasks are waiting for execution.
The Executor (The Muscle): The executor is the mechanism that actually runs the task. Unlike the scheduler, which just monitors, the executor is responsible for spinning up the compute resources (whether that's a process, a Docker container, or a Kubernetes pod) to execute the task code.

Choosing Your Executor

The "architecture" of your Airflow setup is largely defined by your choice of executor. For example, if you use the LocalExecutor, tasks run as sub-processes on the same machine as the scheduler. If you scale to CeleryExecutor or KubernetesExecutor, you move execution off the scheduler machine to a distributed cluster.

# In airflow.cfg, you define the executor:
[core]
executor = CeleryExecutor

When you write a DAG, your code describes what to do, but the Scheduler and Executor determine when and where it happens. By separating the monitoring (Scheduler) from the action (Executor), Airflow ensures that even if a task takes a long time to run or crashes, the scheduler remains free to continue monitoring other tasks in your pipeline.

Key Takeaway: The scheduler identifies tasks ready for work, the metadata database tracks their state, and the executor carries out the actual heavy lifting of task execution.