Course contentsShow
← Machine Learning and Deep Learning
Lesson 2747 of 3,53859. Distributed Training: Data ParallelismFree lesson

Communication Patterns in ZeRO

Understanding all-gather and reduce-scatter collectives used to reconstruct parameters and aggregate gradients.

Communication Patterns in ZeRO

What you'll learn: How ZeRO orchestrates all-gather and reduce-scatter collective operations to reconstruct parameters and aggregate gradients across distributed workers.

The Communication Dance

In ZeRO (Zero Redundancy Optimizer), each GPU holds only a shard of the model's parameters, gradients, and optimizer states. To actually perform forward and backward passes, workers must communicate strategically to temporarily reconstruct what they need, then scatter results back.

All-Gather: Reconstructing Parameters

When a GPU needs the full layer parameters for computation, it uses all-gather. Think of it like assembling a puzzle: each worker contributes their piece (shard), and everyone receives the complete picture. During the forward pass, workers all-gather parameter shards just-in-time, compute their portion of activations, then discard the borrowed parameters to save memory.

Reduce-Scatter: Aggregating Gradients

After the backward pass, each worker has computed gradients for the entire model, but only needs to update its own shard of parameters. Reduce-scatter combines gradients across workers (summing or averaging) and simultaneously distributes the result so each worker receives only the gradient portion corresponding to its parameter shard. It's like collecting everyone's contributions, totaling them up, and giving each person only the slice they're responsible for.

Why This Pattern Matters

This choreography—all-gather before computation, reduce-scatter after gradients—minimizes memory while keeping all workers synchronized. Unlike simple all-reduce (which broadcasts full results everywhere), reduce-scatter is memory-efficient: you get exactly what you own.

Key Takeaway: ZeRO uses all-gather to temporarily reconstruct full parameters for computation and reduce-scatter to aggregate gradients while keeping only each worker's shard, enabling memory-efficient distributed training through carefully orchestrated collective communication.