Declaring Dependencies in Orchestration Tools
You'll learn how to explicitly tell orchestration frameworks which tasks must run before others using their specific syntax patterns.
The Core Idea
Orchestration tools like Airflow and Prefect need you to explicitly declare which tasks depend on which others. This isn't automatic—you must tell the framework "Task B can only start after Task A completes." Different tools offer different syntax patterns, but they all accomplish the same goal: building your DAG (Directed Acyclic Graph) structure in code.
Think of it like giving directions: you're not just listing tasks, you're specifying the order and prerequisites for each step.
Common Syntax Patterns
Airflow offers multiple ways to declare dependencies:
# Bitshift operators (most common)
task_a >> task_b # task_a runs before task_b
task_a >> [task_b, task_c] # task_a before both b and c
[task_a, task_b] >> task_c # both a and b before c
# Explicit methods
task_a.set_downstream(task_b)
task_b.set_upstream(task_a)
Prefect uses a context manager approach:
from prefect import flow, task
@flow
def my_pipeline():
result_a = task_a()
result_b = task_b(result_a) # implicit dependency via parameter
# Dependencies inferred from data flow
Key Patterns
- Sequential chains:
task_a >> task_b >> task_c - Fan-out: One task triggers multiple parallel tasks
- Fan-in: Multiple tasks must complete before one starts
- Conditional dependencies: Some tools support dynamic dependency creation
Key Takeaway: Dependency declaration is how you translate your pipeline's logical flow into executable orchestration code—each tool has its own syntax, but all require you to explicitly map which tasks wait for which predecessors.