Course contentsShow
← Data Science
Lesson 1834 of 2,14543. Data Pipelines and OrchestrationFree lesson

Defining DAGs in Airflow

Creating your first DAG: structure, default_args, schedule intervals, and task definitions.

Defining DAGs in Airflow

You'll learn how to create your first Directed Acyclic Graph (DAG) in Airflow, including structure, configuration, and task definitions.

The Core Structure

A DAG in Airflow is a Python script that defines the workflow blueprint. Think of it like writing a recipe: you specify the ingredients (default settings), cooking schedule (when to run), and step-by-step instructions (tasks). The DAG itself doesn't execute work—it just describes what should happen and when.

Key Components

Default Arguments (default_args)

This dictionary sets common configuration for all tasks in your DAG—owner, retries, retry delays, and start dates. Instead of repeating these settings for each task, you define them once:

default_args = {
    'owner': 'data_team',
    'retries': 2,
    'start_date': datetime(2024, 1, 1)
}

DAG Declaration

You instantiate the DAG object with an identifier, schedule, and defaults:

from airflow import DAG

dag = DAG(
    'my_first_pipeline',
    default_args=default_args,
    schedule_interval='@daily',
    catchup=False
)

Schedule Intervals

The schedule_interval parameter controls when your pipeline runs: '@daily', '@hourly', or cron expressions like '0 2 * * *' (2 AM daily).

Task Definitions

Tasks are the actual work units. Each task inherits the DAG's defaults and represents one operation in your pipeline—extraction, transformation, or loading.

The Analogy

Creating a DAG is like programming a smart thermostat: you set the rules (default_args), decide when it activates (schedule_interval), and define what actions it takes (tasks). The thermostat doesn't heat your home immediately—it just knows the plan.

Key Takeaway: A DAG in Airflow is a Python definition that specifies default configurations, scheduling rules, and task structure without executing anything—it's the blueprint for your data pipeline.