Backfilling and Catchup
What you'll learn: How Airflow handles running DAG instances for past dates and what the catchup parameter controls.
The Problem: Missed Schedules and Historical Data
Imagine you deployed a daily pipeline on Monday, but it should have been running since last Thursday. You now have a gap—three days of unprocessed data. Or suppose your pipeline failed Friday and Saturday, and you need to reprocess those intervals.
This is where backfilling comes in: running historical DAG instances for past schedule intervals that were either skipped or need reprocessing.
How Airflow Handles Catchup
When you define a DAG with a start_date and schedule_interval, Airflow generates one DAG run per interval. By default, Airflow uses catchup=True, meaning when you first deploy or unpause the DAG, it will automatically create runs for every interval between start_date and now.
If your start_date was 30 days ago and you schedule daily runs, Airflow will queue 30 DAG runs immediately—backfilling all those missing intervals.
Controlling Catchup Behavior
Set catchup=False in your DAG definition to disable automatic backfilling. This tells Airflow: "Only run from the next scheduled interval forward; ignore the past."
dag = DAG(
'my_pipeline',
start_date=datetime(2024, 1, 1),
schedule_interval='@daily',
catchup=False # Don't auto-run past intervals
)
Manual Backfilling
Even with catchup=False, you can manually backfill specific date ranges using the Airflow CLI:
airflow dags backfill my_pipeline --start-date 2024-01-01 --end-date 2024-01-07
This gives you precise control over which historical intervals to reprocess.
Key Takeaway: Backfilling reruns DAGs for past intervals; catchup=True automates this on deployment, while catchup=False requires manual backfills for historical data processing.