Course contentsShow
← Machine Learning and Deep Learning
Lesson 2871 of 3,53862. ML Pipelines and OrchestrationFree lesson

Writing Your First Airflow DAG

Creating a simple ML pipeline DAG with PythonOperator, defining task dependencies using >> and << operators.

Writing Your First Airflow DAG

You'll learn how to create a basic Airflow DAG that orchestrates a simple ML pipeline using PythonOperator and task dependencies.

Building Your First DAG

A DAG (Directed Acyclic Graph) in Airflow is like a recipe that specifies what tasks to run and in what order. For ML pipelines, each task might represent a step like data loading, preprocessing, training, or evaluation.

The PythonOperator executes a Python function as a task. You define multiple operators and connect them using dependency operators: >> (downstream) and << (upstream). Think of >> as "then do this next" and << as "do this before."

Core Structure

Here's a simple ML pipeline DAG:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def load_data():
    print("Loading training data...")
    # Your data loading logic

def preprocess():
    print("Preprocessing data...")
    # Your preprocessing logic

def train_model():
    print("Training model...")
    # Your training logic

# Define the DAG
with DAG(
    dag_id='simple_ml_pipeline',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    catchup=False
) as dag:
    
    # Create tasks
    load_task = PythonOperator(
        task_id='load_data',
        python_callable=load_data
    )
    
    preprocess_task = PythonOperator(
        task_id='preprocess',
        python_callable=preprocess
    )
    
    train_task = PythonOperator(
        task_id='train_model',
        python_callable=train_model
    )
    
    # Define dependencies
    load_task >> preprocess_task >> train_task
    # Equivalent: train_task << preprocess_task << load_task

The >> operator creates a clear execution flow: load data, then preprocess, then train. Airflow handles scheduling, retries, and monitoring automatically.

Key Takeaway: An Airflow DAG orchestrates ML pipeline tasks using PythonOperators connected with >> or << operators to define execution order, turning your scattered Python functions into a reliable, scheduled workflow.