In this lesson, you will learn how to make your Airflow DAGs "aware" of the outside world by using sensors to pause execution until specific external conditions or data files are met.
The Concept of Reactive Orchestration
Up until now, we have mostly focused on DAGs that run on rigid schedules—for example, triggering at exactly 8:00 AM every day. However, in modern data engineering, data often arrives at unpredictable times. If your downstream dbt transformations start before the source file hits your S3 bucket or database, your pipeline will fail.
A Sensor is a special type of operator designed to wait for a condition to be met. Instead of executing code and finishing, a sensor "pokes" the environment at a regular interval. If the condition (like checking if a file exists) returns True, the sensor succeeds and the DAG proceeds. If it returns False, the sensor stays in a running state until it either succeeds or hits a timeout.
Implementing a File Sensor
Imagine you have a landing zone for CSV files. You can use the FileSensor to pause your pipeline until the file arrives:
from airflow.sensors.filesystem import FileSensor
wait_for_file = FileSensor(
task_id='wait_for_sales_data',
filepath='/data/inbound/sales_report.csv',
fs_conn_id='fs_default',
poke_interval=60, # Check every 60 seconds
timeout=3600 # Give up after 1 hour
)
# Your dbt transformation task follows the sensor
dbt_run_transformations >> wait_for_file
External Triggers
Beyond waiting for files, you might need to trigger one DAG based on the successful completion of another, potentially residing in a different project. By using the ExternalTaskSensor, you create a logical link between two independent workflows. This allows you to orchestrate complex dependencies across distributed data environments without manually checking if a previous process finished.
By integrating sensors, you transform your static scheduled DAGs into dynamic, event-driven pipelines that react to the reality of your data's availability rather than just the clock.
Key Takeaway: Sensors allow you to build resilient, data-aware pipelines by pausing task execution until external preconditions—such as file arrival or downstream completion—are successfully met.