In this lesson, you will learn how to design data pipelines that remain idle until specific external conditions are met, ensuring your processing tasks only run when the necessary data is verified as available.
The Problem of "Pull" vs. "Push"
In a standard pipeline, tasks might execute on a fixed schedule (time-based). However, external data sources—like an upstream API delivery or a file landing in an S3 bucket—often don't adhere to a strict clock. If you trigger your DAG prematurely, your tasks fail because the data isn't there, leading to unnecessary retries and resource waste.
Enter Sensors
A Sensor is a special type of operator designed to "poll" for a condition. Unlike a standard operator that performs a transformation, a sensor pauses the execution of its branch in the DAG until a specific criterion is met. If the condition is false, the sensor waits (or "sleeps") and re-checks based on a poke_interval.
Consider an example where you need to wait for a specific CSV file to land in an S3 bucket before triggering an ingestion task:
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
wait_for_file = S3KeySensor(
task_id='wait_for_data',
bucket_name='my-data-bucket',
bucket_key='daily_import.csv',
poke_interval=60, # Checks every 60 seconds
timeout=3600 # Fails if data doesn't arrive within an hour
)
# Downstream task only executes after wait_for_data succeeds
process_data = PythonOperator(
task_id='process_data',
python_callable=my_transform_func
)
wait_for_file >> process_data
Why This Matters
By using sensors, you transform your static, time-bound pipelines into event-driven workflows. This creates a "dynamic" feel because the pipeline progresses at the speed of your data, not the speed of your scheduler. It saves compute costs by avoiding idle task slots and prevents the frustration of "data not found" errors in your logs. When implementing these, always define a timeout to ensure that a stuck upstream process doesn't cause your DAG to run indefinitely.
Key Takeaway: Sensors act as intelligent gates that synchronize your pipeline execution with the actual arrival of external data, eliminating the need for brittle, time-based scheduling.