Generating DAGs Programmatically Using Python Loops

In this lesson, you will learn how to scale your orchestration efforts by generating multiple DAGs dynamically from a single configuration file using Python loops.

The Challenge of Redundancy

As your data platform grows, you may find yourself creating dozens of similar DAGs—for example, one for every table in a database that requires the same extraction, loading, and transformation steps. Hardcoding these individually leads to "copy-paste" errors, inconsistent configurations, and a maintenance nightmare whenever you need to update a shared parameter like an email alert or a schedule interval.

The Dynamic Approach

Instead of writing 50 separate files, you can treat your DAG definition as a Python function that is called inside a loop. By storing your pipeline metadata (like source names, schemas, or schedules) in a JSON or YAML file, you can iterate through that configuration to instantiate unique DAG objects programmatically.

Here is how you structure this in your Airflow dags/ folder:

import json
from airflow import DAG
from airflow.operators.bash import BashOperator

# 1. Load configuration metadata
with open('/dags/config/pipelines.json') as f:
    configs = json.load(f)

# 2. Iterate through configurations
for pipeline in configs:
    dag_id = f"process_{pipeline['table_name']}"
    
    # 3. Dynamically instantiate the DAG object
    globals()[dag_id] = DAG(
        dag_id=dag_id,
        schedule_interval=pipeline['schedule'],
        start_date=pipeline['start_date']
    )
    
    # 4. Define tasks within the dynamic context
    with globals()[dag_id]:
        BashOperator(
            task_id="run_sync",
            bash_command=f"echo Syncing {pipeline['table_name']}"
        )

Why It Works

By using the globals() dictionary, you trick the Airflow scheduler into seeing these dynamically generated variables as legitimate DAG objects. Because the DAG definition is logic-based, you can add new pipelines simply by updating your pipelines.json file. This pattern decouples your pipeline configuration from your orchestration logic, ensuring that your codebase remains DRY (Don't Repeat Yourself) even as your infrastructure scales.

Key Takeaway: By looping over configuration files to instantiate DAG objects, you transform your pipeline maintenance from a manual per-file chore into a scalable, data-driven automation process.