In this lesson, you’ll learn how to use Trigger Rules to control exactly when a task executes based on the status of its upstream dependencies.
The Problem with Default Behavior
By default, an Airflow task only runs if all of its direct upstream parents succeed. While this is perfect for linear pipelines, real-world data engineering often requires more nuance. What happens if you have a task that needs to run even if a previous task fails, or perhaps only after any parent finishes? Trigger rules allow you to move beyond the "all-success" default.
Understanding Trigger Rules
A trigger rule is an argument you pass to an operator that defines the condition for the downstream task to start. Some of the most common rules include:
all_success(Default): All parents must complete successfully.all_failed: All parents must have failed.all_done: The task runs as soon as all parents are finished, regardless of whether they succeeded, failed, or were skipped.one_success: The task runs as soon as at least one parent succeeds.none_failed: The task runs if all parents have succeeded or been skipped (but none failed).
Practical Example
Imagine you are building a reporting pipeline. You have a process_data task and a cleanup_temp_files task. Even if the data processing fails, you want to ensure the temp files are deleted to save storage. You can force the cleanup to run using all_done:
from airflow.operators.bash import BashOperator
cleanup_task = BashOperator(
task_id='cleanup_temp_files',
bash_command='rm -rf /tmp/data_processing/*',
trigger_rule='all_done', # Runs even if 'process_data' fails
dag=dag
)
By setting the rule to all_done, you ensure the cleanup logic is decoupled from the success of the upstream operation. This is critical for maintaining infrastructure hygiene in complex workflows where failures are expected during specific stages of data ingestion.
By mastering these rules, you gain granular control over your DAG’s execution flow, ensuring your pipelines remain resilient even when individual components experience unexpected issues.
Key Takeaway: Trigger rules allow you to define execution logic based on upstream task outcomes, enabling you to build fault-tolerant pipelines that handle errors and partial successes gracefully.