FSDP Sharding Strategy Overview
You'll learn how FSDP distributes model parameters across GPUs and reconstructs them just-in-time during training.
The Core Idea
Fully Sharded Data Parallel (FSDP) solves the memory problem of large models by sharding (splitting) the model's parameters, gradients, and optimizer states across all available GPUs. Unlike DDP where every GPU holds a complete copy of the model, FSDP ensures each GPU only stores a fraction of the total parameters.
Think of it like a library where books (parameters) are distributed across multiple shelves (GPUs). When you need to read a specific book, you temporarily gather it from the shelf that holds it, read it, then return it.
How FSDP Works During Training
Forward Pass
When a layer needs to compute, FSDP performs an all-gather operation to temporarily reconstruct the full parameters from all shards across GPUs. The layer executes its computation, then immediately discards the gathered parameters, keeping only its own shard.
Backward Pass
During backpropagation, the same gather-compute-discard pattern occurs. After computing gradients for a layer, FSDP performs a reduce-scatter to distribute gradient shards back to their owning GPUs, where they update only their portion of parameters.
Memory Efficiency
This on-demand gathering means each GPU's peak memory is dramatically lower than holding the entire model. The trade-off is increased communication overhead—you're constantly gathering and scattering data across GPUs during training.
Key Takeaway: FSDP shards parameters across GPUs and uses all-gather operations to temporarily reconstruct them during forward/backward passes, then immediately discards non-owned parameters to minimize memory footprint at the cost of communication overhead.