Course contentsShow
← Machine Learning and Deep Learning
Lesson 2870 of 3,53862. ML Pipelines and OrchestrationFree lesson

Airflow Architecture and Core Concepts

DAGs, operators, executors, and the scheduler—how Airflow organizes and runs workflow tasks across distributed systems.

Airflow Architecture and Core Concepts

What You'll Learn

You'll understand how Apache Airflow organizes and executes ML workflow tasks through DAGs, operators, executors, and schedulers across distributed systems.

Core Architecture Components

Airflow structures workflows as Directed Acyclic Graphs (DAGs), where each node represents a task and edges define dependencies. Think of a DAG as a recipe: each step must happen in order, but some steps can run in parallel if they don't depend on each other.

DAGs: Workflow Blueprints

A DAG defines what to run and when, but doesn't execute tasks itself. It's the master plan for your ML pipeline—data preprocessing → training → validation → deployment.

Operators: Task Definitions

Operators are the building blocks that define individual tasks. Examples include PythonOperator (run Python functions), BashOperator (shell commands), or custom operators for model training. Each operator encapsulates one unit of work.

The Scheduler: Orchestration Brain

The scheduler continuously monitors DAGs, determines which tasks are ready to run (dependencies satisfied), and submits them to executors. It handles the "when" by checking schedules and trigger conditions you've defined.

Executors: Task Runners

Executors determine where and how tasks run. The LocalExecutor runs tasks on the same machine, while CeleryExecutor or KubernetesExecutor distribute tasks across worker nodes—critical for scaling ML workloads. The executor picks up tasks from the scheduler's queue and manages actual execution.

How They Work Together

The scheduler reads your DAG files, creates task instances, and queues ready tasks. The executor picks them up and runs them on available workers. Metadata about runs, task states, and logs flows back to Airflow's database, enabling monitoring and retries.

Key Takeaway: Airflow separates workflow definition (DAGs/operators) from orchestration (scheduler) and execution (executors), enabling scalable, distributed ML pipeline management across heterogeneous infrastructure.