In this lesson, you will learn how to build reactive data pipelines that wait for incoming files in external storage before initiating downstream processing.
Why External Sensors Matter
In many production environments, you don't control the arrival time of source data. Instead of running pipelines on a rigid schedule—which leads to wasted compute if data is late or idle gaps if data arrives early—we use Sensors. A sensor is a special type of operator that polls an external source (like an S3 bucket or a file system) and stays in a "running" state until a specific condition, such as the appearance of a new file, is met.
Implementing the Trigger
By using sensors, you transform your DAG from a clock-based schedule into a data-driven workflow. For example, if you are expecting a daily partition in an S3 bucket, you can use an S3KeySensor. The sensor checks for the existence of that specific file or partition path. If the file exists, the sensor succeeds and allows the DAG to progress to your transformation tasks. If it doesn't, the sensor sleeps and checks again, preventing your pipeline from failing due to missing data.
Consider this basic structure:
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor
# The sensor waits for the specific date partition to appear
wait_for_data = S3KeySensor(
task_id='wait_for_s3_data',
bucket_name='my-data-bucket',
bucket_key='logs/{{ ds }}/data.csv',
timeout=18 * 60 * 60, # Timeout after 18 hours
poke_interval=60 # Check every minute
)
# This downstream task only runs once the file is confirmed present
process_data = PythonOperator(
task_id='process_data',
python_callable=my_processing_func
)
wait_for_data >> process_data
Strategic Design
When designing these triggers, always define a timeout and a poke_interval. Without a timeout, a sensor could run indefinitely, blocking your worker slots. Additionally, sensors are most effective when they target specific partitions (e.g., using Jinja templating like {{ ds }}) to ensure you are processing the correct "window" of data. This ensures your pipelines are not just reactive, but also idempotent and logically aligned with your partition strategy.
Key Takeaway: Sensors shift your orchestration logic from "running at a fixed time" to "running as soon as data arrives," creating efficient, event-driven pipelines that handle external dependencies gracefully.