Course contentsShow
← Machine Learning and Deep Learning
Lesson 2745 of 3,53859. Distributed Training: Data ParallelismFree lesson

ZeRO Stage 2: Gradient Partitioning

Extending ZeRO-1 by also partitioning gradients across devices, further reducing memory footprint.

ZeRO Stage 2: Gradient Partitioning

What You'll Learn

You'll learn how ZeRO Stage 2 extends memory optimization beyond optimizer states by partitioning gradients across devices during distributed training.

The Core Idea

In ZeRO Stage 1, you partitioned optimizer states across devices so each GPU only stores a fraction of the optimizer memory. ZeRO Stage 2 takes this further: it also partitions gradients.

Here's what happens during backward pass:

  1. Computation phase: Each device still computes its full set of gradients locally during backpropagation
  2. Reduction phase: Instead of keeping all gradients on all devices (as in standard DDP), gradients are reduced only to their designated "owner" device
  3. Storage phase: Each device only stores the gradients for the parameters whose optimizer states it owns

Think of it like a collaborative note-taking team: everyone takes notes during the lecture (computes gradients), but then each person only keeps the notes for their assigned chapters (partitioned gradients), not the entire textbook.

Memory Impact

Since gradients typically match the model size in memory footprint, ZeRO Stage 2 cuts gradient memory by a factor of N (number of devices). Combined with Stage 1's optimizer partitioning, you're now saving memory on two major components instead of one.

The trade-off? Slightly more sophisticated communication patterns during the reduce-scatter operations, but the memory savings often enable training models that wouldn't otherwise fit.

Key Takeaway: ZeRO Stage 2 partitions gradients across devices in addition to optimizer states, reducing per-device memory by another 1/N factor while maintaining mathematical equivalence to non-partitioned training.