Course contentsShow
← Data Science
Lesson 1844 of 2,14543. Data Pipelines and OrchestrationFree lesson

Dynamic Dependencies

Creating dependencies that are determined at runtime based on data availability, branching logic, or external conditions.

Dynamic Dependencies

What you'll learn: How to create pipeline dependencies that change at runtime based on data, branching logic, or external conditions rather than being fixed in advance.

The Core Idea

In most data pipelines, you declare dependencies ahead of time: "Task B runs after Task A completes." But what if you don't know which tasks need to run until after you've looked at the data?

Dynamic dependencies let your pipeline decide its own structure while running. Think of it like a "choose your own adventure" book—the next chapter depends on choices made along the way, not a fixed table of contents.

Common Scenarios

Data availability: You might have a task that checks which data sources updated today, then spawns processing tasks only for those sources.

Branching logic: After analyzing a dataset, you might trigger different validation pipelines depending on data quality scores or record counts.

External conditions: A task queries an API to see which regions have new data, then dynamically creates one downstream task per region.

How It Works Conceptually

Instead of declaring task_b.set_upstream(task_a) for every possible path, you write logic that generates tasks and dependencies during execution. The orchestrator evaluates this logic when it reaches that point in the DAG, then builds the appropriate downstream graph on the fly.

This is particularly powerful for pipelines that process variable numbers of entities (customers, products, regions) or that need to adapt to changing upstream conditions without manual DAG rewrites.

Key Takeaway: Dynamic dependencies allow your pipeline structure to adapt at runtime based on data content or external conditions, making pipelines more flexible and reducing the need for complex pre-planning of every possible execution path.