In this lesson, you will learn how to configure and execute Airflow’s catchup mechanism to reliably process historical data intervals after updating your DAG logic.
When you modify your transformation logic—such as adding a new column to a dbt model or changing a business rule—the data generated before that change remains in its stale state. To fix this, we use backfilling, which is the process of re-executing your DAG for specific historical time ranges to ensure your data warehouse reflects the latest logic.
The Catchup Mechanism
In Airflow, the catchup parameter (found in your DAG object) controls whether the scheduler should automatically trigger instances for every interval between the start_date and the current date.
dag = DAG(
'my_historical_pipeline',
start_date=datetime(2023, 1, 1),
catchup=True, # Set to True to backfill missed intervals
schedule_interval='@daily'
)
By default, catchup is often True. When you deploy a new DAG with a start_date in the past, Airflow will immediately begin scheduling all missing daily runs. While this is powerful for historical consistency, it can overwhelm your infrastructure if not managed carefully.
Executing Targeted Backfills
If you want more control, or if you need to re-run only a specific window after a code deployment, you should set catchup=False in your DAG definition and use the CLI to trigger a manual backfill. This allows you to process data in batches without triggering a flood of tasks:
airflow dags backfill -s 2023-01-01 -e 2023-01-10 my_historical_pipeline
This command tells Airflow to execute the DAG for the range starting January 1st to January 10th. Because we previously covered idempotency (Lesson 9), you can run these backfills safely, knowing that your transformation tasks are designed to overwrite existing data for a given execution date without creating duplicates or corrupting the final state. By combining idempotent tasks with manual backfilling, you ensure your data warehouse remains a "single source of truth," regardless of when you updated your pipeline code.
Key Takeaway: Catchup automates historical processing, but manual backfilling provides the precise control needed to rebuild data sets after logic changes while maintaining system stability.