DVC Pipelines and Stages
What you'll learn: How to build reproducible, trackable data science workflows by defining pipelines as discrete stages with explicit dependencies and outputs.
The Core Idea
Think of a DVC pipeline like a recipe with explicit ingredient lists and step-by-step instructions. Each stage is one step (like "preprocess data" or "train model"), and you declare exactly what files it needs (dependencies) and what it produces (outputs). DVC tracks all of this, making your entire workflow reproducible with a single command.
Unlike running scripts manually or relying on documentation, DVC pipelines turn your workflow into code that anyone can replay identically.
How Stages Work
A stage defines:
- Command: The script or code to run
- Dependencies: Input files, datasets, or code it needs
- Outputs: Files it creates (datasets, models, metrics)
You define stages in dvc.yaml. When you run dvc repro, DVC automatically executes only the stages whose dependencies have changed—skipping unnecessary work.
Example Structure
stages:
preprocess:
cmd: python preprocess.py
deps:
- data/raw.csv
- preprocess.py
outs:
- data/processed.csv
train:
cmd: python train.py
deps:
- data/processed.csv
- train.py
outs:
- models/model.pkl
Here, train depends on preprocess's output. Change raw.csv, and both stages rerun. Change only train.py, and only training reruns.
Why This Matters
DVC stages create end-to-end tracking: from raw data through transformations to final models. You get reproducibility (anyone can recreate results), efficiency (only rerun what changed), and transparency (the pipeline structure documents your workflow).
Key Takeaway: DVC pipelines define reproducible workflows as stages with explicit dependencies and outputs, enabling automatic tracking and selective re-execution of your entire data science process.