DeepSpeed ZeRO Stage Selection
You'll learn how to choose the right ZeRO stage based on your model's memory needs and communication constraints.
Understanding the Three Stages
When training large models with DeepSpeed, you must decide which ZeRO stage to use. Each stage offers a different balance between memory savings and communication overhead:
ZeRO Stage 1 partitions only the optimizer states across GPUs. It's the lightest option—minimal communication overhead but modest memory savings (roughly 4x reduction in optimizer memory). Choose this when your model almost fits in memory and you want to avoid extra communication costs.
ZeRO Stage 2 partitions both optimizer states and gradients. This roughly doubles the memory savings of Stage 1 (8x total reduction). The communication overhead increases slightly since gradients must be gathered during the backward pass. Use this for medium-large models where Stage 1 isn't enough but full parameter sharding feels excessive.
ZeRO Stage 3 partitions everything: optimizer states, gradients, and model parameters. This provides maximum memory efficiency, enabling training of truly massive models. However, parameters must be gathered before every forward pass and re-sharded afterward, creating the highest communication overhead. Choose this when your model simply won't fit otherwise—memory constraints force your hand.
The Decision Framework
Think of it like packing for a trip: Stage 1 is sharing toiletries (optimizer states), Stage 2 adds sharing clothes (gradients), and Stage 3 means sharing everything including your tent (parameters). Each step saves luggage space but requires more coordination.
Key factors:
- Model size: <1B parameters → Stage 1, 1-10B → Stage 2, 10B+ → Stage 3
- Network bandwidth: Fast interconnect (InfiniBand) tolerates Stage 3 better
- GPU memory: If training fails with OOM, move up one stage
Key Takeaway: Start with the lowest ZeRO stage that fits your model in memory—Stage 1 for maximum speed, Stage 2 for balance, Stage 3 when memory is critical and you have adequate network bandwidth.