ZeRO Stage Decomposition Concepts
What you'll learn: How ZeRO divides model training memory into three distinct components that can be sharded across GPUs.
The Three Memory Components
When training a neural network with standard optimizers like Adam, memory consumption comes from three main sources:
- Optimizer States: Momentum and variance buffers (for Adam: 2× parameters)
- Gradients: One gradient tensor per parameter (1× parameters)
- Parameters: The model weights themselves (1× parameters)
For a model with P parameters using mixed precision and Adam, this totals roughly 16 bytes per parameter (4 + 4 + 8), making memory the primary bottleneck for large models.
Why This Matters for ZeRO
Traditional Data Parallel (DDP) training replicates all three components on every GPU. Each worker holds a complete copy of optimizer states, gradients, and parameters. This is wasteful—why should 8 GPUs each store identical optimizer states?
ZeRO's insight: These three components can be partitioned (sharded) across workers, with each GPU responsible for only a fraction of each.
The Stage Hierarchy
ZeRO defines three progressive stages of sharding:
- Stage 1: Shard optimizer states only (~4× memory reduction)
- Stage 2: Shard optimizer states + gradients (~8× reduction)
- Stage 3: Shard optimizer states + gradients + parameters (~N× reduction for N GPUs)
Each stage builds on the previous, trading increased communication for reduced per-GPU memory.
Think of it like a library: instead of everyone owning every book (DDP), you share a collection where each person keeps certain volumes and borrows others when needed (ZeRO).
Key Takeaway: ZeRO decomposes training memory into optimizer states, gradients, and parameters—three independently shardable components that enable training massive models by distributing memory burden across GPUs.