Directed Acyclic Graphs (DAGs)
You'll learn how ML pipelines are structured as DAGs where nodes represent tasks and edges define their dependencies.
What is a DAG?
A Directed Acyclic Graph (DAG) is the mathematical structure underlying ML pipeline orchestration. In this representation:
- Nodes are individual tasks (data loading, preprocessing, training, evaluation, deployment)
- Edges are directed arrows showing which task must complete before another can start
- Acyclic means no circular dependencies—you can't have Task A depending on Task B, which depends on Task C, which depends back on Task A
Why DAGs for ML Pipelines?
Think of a DAG like a recipe flowchart. You can't frost a cake before baking it, and you can't bake it before mixing ingredients. Some steps (like preheating the oven and mixing ingredients) can happen in parallel since they don't depend on each other.
In ML:
- Data validation must complete before preprocessing
- Preprocessing must finish before training
- Training must complete before evaluation
- But data validation and environment setup can run in parallel
The DAG explicitly captures these dependencies, allowing orchestration tools to:
- Execute independent tasks in parallel for efficiency
- Ensure tasks run in valid order
- Retry failed tasks without rerunning successful ones
- Visualize the entire pipeline structure
The orchestration system walks through the DAG, executing each node only when all its parent nodes (dependencies) have succeeded.
Key Takeaway: DAGs provide the formal structure for ML pipelines, where nodes are discrete tasks and directed edges define execution order, enabling automated orchestration, parallelization, and dependency management.