Course contentsShow
← Machine Learning and Deep Learning
Lesson 2732 of 3,53859. Distributed Training: Data ParallelismFree lesson

All-Gather and Reduce-Scatter Operations

The core collective communication primitives FSDP uses to gather full parameters and distribute gradients.

All-Gather and Reduce-Scatter Operations

You'll learn the two fundamental collective communication patterns that FSDP uses to orchestrate parameter gathering and gradient distribution across workers.

The Core Problem

In FSDP, each GPU holds only a shard (portion) of the model's parameters. Before running a forward or backward pass, you need the complete parameters for certain layers. After computing gradients, you need to distribute them back to their respective owners. Two operations make this possible.

All-Gather: Assembling the Whole

All-Gather collects sharded data from all workers and reconstructs the complete tensor on every GPU.

Analogy: Imagine a jigsaw puzzle where each person holds one piece. All-Gather is when everyone photocopies their piece and shares it with everyone else, so everyone ends up with the complete puzzle.

In FSDP, before computing a layer's forward pass, an All-Gather collects that layer's weight shards from all GPUs, temporarily reconstructing the full parameter tensor on each device. After computation, this full tensor is typically discarded to save memory.

Reduce-Scatter: Distributing Aggregated Gradients

Reduce-Scatter first reduces (sums) contributions from all workers, then scatters different portions of the result to different GPUs.

Analogy: Each person calculates part of a group expense. Reduce-Scatter totals everyone's contributions, then assigns each person responsibility for tracking one category of the final budget.

During backward pass, gradients are computed for the full parameter tensor. Reduce-Scatter sums these gradients across all workers, then distributes each shard back to its owner GPU for the optimizer update.

Why These Primitives?

Together, All-Gather and Reduce-Scatter minimize memory while maintaining mathematical correctness—each GPU only stores its parameter shard permanently but can temporarily access full parameters when needed.

Key Takeaway: All-Gather reconstructs full parameters from shards for computation; Reduce-Scatter aggregates and redistributes gradients back to shard owners—these are the heartbeat operations of FSDP's memory-efficient distributed training.