Memory vs Communication Tradeoffs
You'll learn: How each ZeRO stage balances memory savings against communication overhead, and when to choose which stage.
The Core Tradeoff
Every ZeRO stage reduces memory consumption by partitioning different model components (optimizer states, gradients, parameters) across workers. But there's no free lunch: the more you partition, the more communication you need during training to gather the pieces when they're needed.
The Three-Stage Spectrum
ZeRO Stage 1 (optimizer partitioning) gives modest memory savings with minimal communication overhead. Each worker only needs to all-gather optimizer states during the optimizer step—not during forward or backward passes. This is ideal when you're close to fitting in memory but need a small boost.
ZeRO Stage 2 (optimizer + gradient partitioning) provides better memory reduction but adds a reduce-scatter operation during the backward pass to distribute gradient shards. The communication increase is moderate, making this the sweet spot for many large models that don't quite fit with DDP.
ZeRO Stage 3 (full parameter partitioning) delivers maximum memory savings by sharding even the model parameters. However, it requires all-gather operations before every forward and backward layer computation, significantly increasing communication volume. Use this when memory is severely constrained and you're willing to accept slower iteration times.
When to Use Each
- Stage 1: Model fits in memory but optimizer states push you over
- Stage 2: Gradients + optimizer states are your bottleneck; balanced approach
- Stage 3: Model parameters themselves don't fit; accept communication cost for trainability
Key Takeaway: ZeRO stages form a spectrum where higher stages trade increasingly frequent communication for progressively larger memory savings—choose based on your memory constraints and network bandwidth.