Course contentsShow
← Data Science
Lesson 1874 of 2,14543. Data Pipelines and OrchestrationFree lesson

DVC Pipelines and Stages

Define reproducible data pipelines with DVC stages, dependencies, and outputs for end-to-end tracking.

DVC Pipelines and Stages

What you'll learn: How to build reproducible, trackable data science workflows by defining pipelines as discrete stages with explicit dependencies and outputs.

The Core Idea

Think of a DVC pipeline like a recipe with explicit ingredient lists and step-by-step instructions. Each stage is one step (like "preprocess data" or "train model"), and you declare exactly what files it needs (dependencies) and what it produces (outputs). DVC tracks all of this, making your entire workflow reproducible with a single command.

Unlike running scripts manually or relying on documentation, DVC pipelines turn your workflow into code that anyone can replay identically.

How Stages Work

A stage defines:

  • Command: The script or code to run
  • Dependencies: Input files, datasets, or code it needs
  • Outputs: Files it creates (datasets, models, metrics)

You define stages in dvc.yaml. When you run dvc repro, DVC automatically executes only the stages whose dependencies have changed—skipping unnecessary work.

Example Structure

stages:
  preprocess:
    cmd: python preprocess.py
    deps:
      - data/raw.csv
      - preprocess.py
    outs:
      - data/processed.csv
  
  train:
    cmd: python train.py
    deps:
      - data/processed.csv
      - train.py
    outs:
      - models/model.pkl

Here, train depends on preprocess's output. Change raw.csv, and both stages rerun. Change only train.py, and only training reruns.

Why This Matters

DVC stages create end-to-end tracking: from raw data through transformations to final models. You get reproducibility (anyone can recreate results), efficiency (only rerun what changed), and transparency (the pipeline structure documents your workflow).

Key Takeaway: DVC pipelines define reproducible workflows as stages with explicit dependencies and outputs, enabling automatic tracking and selective re-execution of your entire data science process.