Defining Airflow DAGs and Global Configuration

Learners will create a basic Directed Acyclic Graph structure and configure essential parameters like schedule intervals and default task arguments.

In this lesson, you will learn how to structure a basic Airflow DAG and apply global configurations to ensure your workflows remain consistent and maintainable.

Defining the DAG Structure

An Airflow DAG (Directed Acyclic Graph) is a collection of tasks organized to show their relationships and dependencies. Instead of writing monolithic scripts, we define these relationships in a Python file. At the heart of every DAG is the DAG object, which acts as the container for your entire pipeline.

To start, you need to provide a dag_id (a unique identifier) and a start_date. By initializing the DAG using a with statement, any operators you instantiate inside that block are automatically assigned to that specific DAG.

Configuring Default Arguments

Rather than repeating configurations for every single task, Airflow allows you to define default_args. This is a dictionary where you set global parameters like retries, retry_delay, and email_on_failure. By passing these into the DAG constructor, you ensure that every task inherits these settings by default, keeping your code clean and standardized.

Consider this basic structure:

from airflow import DAG
from datetime import datetime, timedelta

default_args = {
    'owner': 'data_eng_team',
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    dag_id='my_first_pipeline',
    default_args=default_args,
    schedule_interval='@daily',
    start_date=datetime(2023, 1, 1),
    catchup=False
) as dag:
    # Tasks go here
    pass

Scheduling and Global Behavior

The schedule_interval defines how often your pipeline runs. Using presets like @daily or Cron expressions gives you precise control over your data freshness. Additionally, setting catchup=False is a vital global configuration; it tells Airflow not to trigger backfill runs for any intervals between the start_date and the current time, which is usually desired when deploying a new pipeline.

By centralizing these settings, you ensure that your orchestration logic is predictable and that global policies—like how many times a task should retry—are applied uniformly across your entire workflow.

Key Takeaway: Use default_args to centralize task configurations and ensure your DAG container acts as the single source of truth for your pipeline's schedule and execution behavior.