All-Gather and Reduce-Scatter Operations
You'll learn the two fundamental collective communication patterns that FSDP uses to orchestrate parameter gathering and gradient distribution across workers.
The Core Problem
In FSDP, each GPU holds only a shard (portion) of the model's parameters. Before running a forward or backward pass, you need the complete parameters for certain layers. After computing gradients, you need to distribute them back to their respective owners. Two operations make this possible.
All-Gather: Assembling the Whole
All-Gather collects sharded data from all workers and reconstructs the complete tensor on every GPU.
Analogy: Imagine a jigsaw puzzle where each person holds one piece. All-Gather is when everyone photocopies their piece and shares it with everyone else, so everyone ends up with the complete puzzle.
In FSDP, before computing a layer's forward pass, an All-Gather collects that layer's weight shards from all GPUs, temporarily reconstructing the full parameter tensor on each device. After computation, this full tensor is typically discarded to save memory.
Reduce-Scatter: Distributing Aggregated Gradients
Reduce-Scatter first reduces (sums) contributions from all workers, then scatters different portions of the result to different GPUs.
Analogy: Each person calculates part of a group expense. Reduce-Scatter totals everyone's contributions, then assigns each person responsibility for tracking one category of the final budget.
During backward pass, gradients are computed for the full parameter tensor. Reduce-Scatter sums these gradients across all workers, then distributes each shard back to its owner GPU for the optimizer update.
Why These Primitives?
Together, All-Gather and Reduce-Scatter minimize memory while maintaining mathematical correctness—each GPU only stores its parameter shard permanently but can temporarily access full parameters when needed.
Key Takeaway: All-Gather reconstructs full parameters from shards for computation; Reduce-Scatter aggregates and redistributes gradients back to shard owners—these are the heartbeat operations of FSDP's memory-efficient distributed training.