Course contentsShow
← Machine Learning and Deep Learning
Lesson 2874 of 3,53862. ML Pipelines and OrchestrationFree lesson

Airflow Scheduling and Triggers

Cron expressions, schedule intervals, catchup behavior, and sensor operators for event-driven pipeline execution.

Airflow Scheduling and Triggers

You'll learn how to control when and why your ML pipelines execute using cron expressions, schedule intervals, catchup behavior, and sensor operators.

Understanding Airflow Scheduling

Airflow gives you fine-grained control over when your DAGs run. Unlike manually triggering pipelines, scheduling automates execution based on time intervals or external events.

Cron expressions are the classic way to define recurring schedules. A cron string like 0 2 * * * means "run at 2 AM daily." Airflow uses these in the schedule_interval parameter of your DAG definition. For ML workflows, you might schedule model retraining weekly (0 0 * * 0) or data validation hourly (0 * * * *).

Schedule intervals can also use Airflow's built-in presets like @daily, @weekly, or timedelta objects for flexibility. Think of these as recurring calendar reminders for your pipeline.

Catchup Behavior

When you deploy a DAG with a start_date in the past, catchup determines whether Airflow runs all missed intervals. If catchup=True, Airflow backfills historical runs—useful for reprocessing training data across time periods. Set catchup=False to skip backfilling and only run forward from deployment time.

Event-Driven Execution with Sensors

Not all pipelines should run on a schedule—sometimes they need to react to events. Sensor operators continuously check for conditions before allowing downstream tasks to execute.

For example, an S3KeySensor waits until new training data lands in a bucket before triggering feature engineering. A FileSensor monitors for annotation files before starting model evaluation. Sensors poll at intervals, acting like patient guards that only open the gate when conditions are met—perfect for event-driven ML workflows where data arrival is unpredictable.

Key Takeaway: Airflow scheduling (cron, intervals, catchup) automates time-based pipeline execution, while sensor operators enable event-driven workflows that react to external conditions like data availability.