Defining DAGs in Airflow
You'll learn how to create your first Directed Acyclic Graph (DAG) in Airflow, including structure, configuration, and task definitions.
The Core Structure
A DAG in Airflow is a Python script that defines the workflow blueprint. Think of it like writing a recipe: you specify the ingredients (default settings), cooking schedule (when to run), and step-by-step instructions (tasks). The DAG itself doesn't execute work—it just describes what should happen and when.
Key Components
Default Arguments (default_args)
This dictionary sets common configuration for all tasks in your DAG—owner, retries, retry delays, and start dates. Instead of repeating these settings for each task, you define them once:
default_args = {
'owner': 'data_team',
'retries': 2,
'start_date': datetime(2024, 1, 1)
}
DAG Declaration
You instantiate the DAG object with an identifier, schedule, and defaults:
from airflow import DAG
dag = DAG(
'my_first_pipeline',
default_args=default_args,
schedule_interval='@daily',
catchup=False
)
Schedule Intervals
The schedule_interval parameter controls when your pipeline runs: '@daily', '@hourly', or cron expressions like '0 2 * * *' (2 AM daily).
Task Definitions
Tasks are the actual work units. Each task inherits the DAG's defaults and represents one operation in your pipeline—extraction, transformation, or loading.
The Analogy
Creating a DAG is like programming a smart thermostat: you set the rules (default_args), decide when it activates (schedule_interval), and define what actions it takes (tasks). The thermostat doesn't heat your home immediately—it just knows the plan.
Key Takeaway: A DAG in Airflow is a Python definition that specifies default configurations, scheduling rules, and task structure without executing anything—it's the blueprint for your data pipeline.