ZeRO Stage 1: Optimizer State Partitioning
You'll learn how ZeRO-1 reduces memory by distributing optimizer states across GPUs while keeping model parameters replicated.
The Core Problem
In standard data parallel training, each GPU stores a complete copy of the model parameters, gradients, and optimizer states (momentum buffers, variance estimates for Adam, etc.). For large models, optimizer states can consume 2-4× more memory than the parameters themselves. With an Adam optimizer and a billion-parameter model, that's roughly 8-12 GB just for optimizer states on every GPU—massive redundancy.
How ZeRO Stage 1 Works
ZeRO-1 keeps model parameters replicated across all GPUs (just like DDP), but partitions the optimizer states. Each GPU only stores and updates the optimizer states for a subset of parameters.
Think of it like a restaurant where every chef has the full recipe book (parameters), but each chef is responsible for storing ingredient prep notes (optimizer states) for only their assigned dishes.
During training:
- Forward and backward passes work identically to DDP
- After gradient all-reduce, each GPU updates only its assigned parameter slice using its local optimizer states
- An all-gather operation broadcasts updated parameters back to all GPUs
The Memory Win
With 8 GPUs, you reduce optimizer state memory by ~8×. For model parameters and gradients, memory usage stays the same as DDP. This is the gentlest ZeRO stage—easy to implement with minimal communication overhead compared to the memory savings.
Key Takeaway: ZeRO Stage 1 partitions optimizer states across GPUs, reducing per-GPU memory by distributing what was previously redundantly stored, while maintaining DDP-like simplicity and performance.