FSDP Forward Pass Mechanics
What you'll learn: How FSDP orchestrates parameter gathering, computation, and shard cleanup during a single forward pass through a distributed model.
The Choreography of a Sharded Forward Pass
In Fully Sharded Data Parallel (FSDP), each GPU holds only a shard of each layer's parameters. When it's time to compute, the forward pass becomes a carefully coordinated dance of gathering, computing, and discarding.
Step-by-Step Execution
1. Pre-forward state: Your GPU holds only 1/N of a layer's parameters (where N is the number of GPUs).
2. All-gather operation: Just before computing that layer, FSDP triggers an all-gather across all GPUs in the process group. Each GPU contributes its shard, and everyone receives the complete, full-precision parameter tensor. Now you temporarily have 100% of the weights.
3. Compute the layer: With full parameters materialized in memory, the GPU performs the standard forward computation (matrix multiplications, activations, etc.) as if this were single-GPU training.
4. Discard unneeded shards: Immediately after computation completes, FSDP frees the parameter shards from other GPUs. Your device retains only its own 1/N shard, drastically reducing memory footprint.
5. Repeat per layer: This gather→compute→discard cycle repeats for each FSDP-wrapped layer as activations flow through the network.
The Memory Trade-off
Think of it like a library system: you don't own all books (parameters), but you can temporarily borrow the complete set when needed, then return what you don't own. The cost is communication time; the benefit is fitting models that would otherwise exceed GPU memory.
Key Takeaway: FSDP's forward pass uses all-gather to temporarily materialize full parameters for computation, then immediately discards non-owned shards—trading communication overhead for massive memory savings layer by layer.