Writing Your First Airflow DAG
You'll learn how to create a basic Airflow DAG that orchestrates a simple ML pipeline using PythonOperator and task dependencies.
Building Your First DAG
A DAG (Directed Acyclic Graph) in Airflow is like a recipe that specifies what tasks to run and in what order. For ML pipelines, each task might represent a step like data loading, preprocessing, training, or evaluation.
The PythonOperator executes a Python function as a task. You define multiple operators and connect them using dependency operators: >> (downstream) and << (upstream). Think of >> as "then do this next" and << as "do this before."
Core Structure
Here's a simple ML pipeline DAG:
from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime
def load_data():
print("Loading training data...")
# Your data loading logic
def preprocess():
print("Preprocessing data...")
# Your preprocessing logic
def train_model():
print("Training model...")
# Your training logic
# Define the DAG
with DAG(
dag_id='simple_ml_pipeline',
start_date=datetime(2024, 1, 1),
schedule_interval='@daily',
catchup=False
) as dag:
# Create tasks
load_task = PythonOperator(
task_id='load_data',
python_callable=load_data
)
preprocess_task = PythonOperator(
task_id='preprocess',
python_callable=preprocess
)
train_task = PythonOperator(
task_id='train_model',
python_callable=train_model
)
# Define dependencies
load_task >> preprocess_task >> train_task
# Equivalent: train_task << preprocess_task << load_task
The >> operator creates a clear execution flow: load data, then preprocess, then train. Airflow handles scheduling, retries, and monitoring automatically.
Key Takeaway: An Airflow DAG orchestrates ML pipeline tasks using PythonOperators connected with >> or << operators to define execution order, turning your scattered Python functions into a reliable, scheduled workflow.