Course contentsShow
← Machine Learning and Deep Learning
Lesson 2730 of 3,53859. Distributed Training: Data ParallelismFree lesson

ZeRO Stage Decomposition Concepts

Understanding optimizer states, gradients, and parameters as the three memory components ZeRO shards.

ZeRO Stage Decomposition Concepts

What you'll learn: How ZeRO divides model training memory into three distinct components that can be sharded across GPUs.

The Three Memory Components

When training a neural network with standard optimizers like Adam, memory consumption comes from three main sources:

  1. Optimizer States: Momentum and variance buffers (for Adam: 2× parameters)
  2. Gradients: One gradient tensor per parameter (1× parameters)
  3. Parameters: The model weights themselves (1× parameters)

For a model with P parameters using mixed precision and Adam, this totals roughly 16 bytes per parameter (4 + 4 + 8), making memory the primary bottleneck for large models.

Why This Matters for ZeRO

Traditional Data Parallel (DDP) training replicates all three components on every GPU. Each worker holds a complete copy of optimizer states, gradients, and parameters. This is wasteful—why should 8 GPUs each store identical optimizer states?

ZeRO's insight: These three components can be partitioned (sharded) across workers, with each GPU responsible for only a fraction of each.

The Stage Hierarchy

ZeRO defines three progressive stages of sharding:

  • Stage 1: Shard optimizer states only (~4× memory reduction)
  • Stage 2: Shard optimizer states + gradients (~8× reduction)
  • Stage 3: Shard optimizer states + gradients + parameters (~N× reduction for N GPUs)

Each stage builds on the previous, trading increased communication for reduced per-GPU memory.

Think of it like a library: instead of everyone owning every book (DDP), you share a collection where each person keeps certain volumes and borrows others when needed (ZeRO).

Key Takeaway: ZeRO decomposes training memory into optimizer states, gradients, and parameters—three independently shardable components that enable training massive models by distributing memory burden across GPUs.