Course contentsShow
← Data Science
Lesson 1837 of 2,14543. Data Pipelines and OrchestrationFree lesson

Backfilling and Catchup

Running historical DAG instances, catchup parameter, and handling missed schedule intervals.

Backfilling and Catchup

What you'll learn: How Airflow handles running DAG instances for past dates and what the catchup parameter controls.

The Problem: Missed Schedules and Historical Data

Imagine you deployed a daily pipeline on Monday, but it should have been running since last Thursday. You now have a gap—three days of unprocessed data. Or suppose your pipeline failed Friday and Saturday, and you need to reprocess those intervals.

This is where backfilling comes in: running historical DAG instances for past schedule intervals that were either skipped or need reprocessing.

How Airflow Handles Catchup

When you define a DAG with a start_date and schedule_interval, Airflow generates one DAG run per interval. By default, Airflow uses catchup=True, meaning when you first deploy or unpause the DAG, it will automatically create runs for every interval between start_date and now.

If your start_date was 30 days ago and you schedule daily runs, Airflow will queue 30 DAG runs immediately—backfilling all those missing intervals.

Controlling Catchup Behavior

Set catchup=False in your DAG definition to disable automatic backfilling. This tells Airflow: "Only run from the next scheduled interval forward; ignore the past."

dag = DAG(
    'my_pipeline',
    start_date=datetime(2024, 1, 1),
    schedule_interval='@daily',
    catchup=False  # Don't auto-run past intervals
)

Manual Backfilling

Even with catchup=False, you can manually backfill specific date ranges using the Airflow CLI:

airflow dags backfill my_pipeline --start-date 2024-01-01 --end-date 2024-01-07

This gives you precise control over which historical intervals to reprocess.

Key Takeaway: Backfilling reruns DAGs for past intervals; catchup=True automates this on deployment, while catchup=False requires manual backfills for historical data processing.