In this lesson, you will learn how to use the expand() method to create multiple task instances at runtime based on the dynamic output of upstream data processes.
Beyond Static DAGs
Up until now, you have likely defined task dependencies as a static structure. If you needed to process ten different files, you might have written ten individual tasks or used a Python loop to generate them when the DAG parses. However, what if the number of files isn't known until your pipeline is actually running?
This is where Dynamic Task Mapping becomes essential. Instead of defining the number of tasks in the DAG file itself, you use expand() to tell Airflow to generate task instances dynamically based on an upstream list or dictionary.
Using expand()
When you use the TaskFlow API, you can take the output of one task—like a list of IDs or file paths—and pass it directly into the expand() method of a downstream task. Airflow will then spawn a separate parallel task instance for every item in that list.
Consider this example where we process a list of regions:
@task
def get_regions():
return ["US", "EU", "APAC"] # This could come from a database or API
@task
def process_region(region):
print(f"Processing data for {region}")
# Use expand to create one task per region
regions = get_regions()
process_region.expand(region=regions)
In this setup, process_region does not run just once. Airflow looks at the result of get_regions(), sees three elements, and triggers three parallel instances of process_region. Each instance operates independently, meaning if the "EU" process fails, the "US" and "APAC" instances can still succeed or be retried individually.
Why use this over loops?
While programmatic loops (which you've learned previously) define tasks at parse time, expand() defines them at runtime. This allows your pipeline to be truly data-driven: your DAG structure adapts to the volume and nature of the data it discovers mid-execution, rather than relying on a hard-coded configuration.
Key Takeaway: Dynamic Task Mapping with expand() enables your pipeline to scale horizontally by spawning task instances based on runtime data, providing granular control and improved fault tolerance for parallel processing.