FSDP Backward Pass and Gradient Sharding
What You'll Learn
You'll understand how FSDP computes gradients locally during backpropagation, synchronizes them across all ranks, and then intelligently retains only each rank's assigned shard to minimize memory overhead.
The Backward Pass Flow
During FSDP's backward pass, gradients are computed layer-by-layer in reverse order. Here's the elegant dance that happens:
1. Local Gradient Computation
Each rank computes gradients for the full parameter set it temporarily reconstructed during the forward pass. Initially, every rank has complete gradients for the parameters it's currently processing—but these gradients are computed from different data batches.
2. Gradient Reduction via Reduce-Scatter
Since each rank trained on different data, their gradients differ. FSDP performs a reduce-scatter operation: gradients are summed (averaged) across all ranks, and each rank receives only the reduced gradient for its assigned parameter shard. This is the inverse of the all-gather used in the forward pass.
Analogy: Imagine a restaurant where each chef makes different dishes (data batches). They all write feedback about the same recipe book (parameters). Instead of every chef keeping the entire feedback document, they collectively merge notes and each chef keeps only the section they're responsible for updating.
3. Retain Only Local Shard
After reduction, each rank discards gradient information for parameters it doesn't own. Only the gradient shard corresponding to its parameter shard is kept in memory, dramatically reducing per-rank memory consumption—just like the parameter sharding itself.
This approach ensures gradients follow the same sharding pattern as parameters, maintaining memory efficiency throughout training while guaranteeing correct distributed optimization.
Key Takeaway: FSDP's backward pass computes full gradients locally, performs a reduce-scatter to average them across ranks, and retains only the gradient shard matching each rank's parameter ownership—keeping memory footprint minimal while preserving training correctness.