Course contentsShow
← Machine Learning and Deep Learning
Lesson 2733 of 3,53859. Distributed Training: Data ParallelismFree lesson

FSDP Forward Pass Mechanics

Step-by-step execution of all-gather to materialize full parameters, compute, then discard unneeded shards.

FSDP Forward Pass Mechanics

What you'll learn: How FSDP orchestrates parameter gathering, computation, and shard cleanup during a single forward pass through a distributed model.

The Choreography of a Sharded Forward Pass

In Fully Sharded Data Parallel (FSDP), each GPU holds only a shard of each layer's parameters. When it's time to compute, the forward pass becomes a carefully coordinated dance of gathering, computing, and discarding.

Step-by-Step Execution

1. Pre-forward state: Your GPU holds only 1/N of a layer's parameters (where N is the number of GPUs).

2. All-gather operation: Just before computing that layer, FSDP triggers an all-gather across all GPUs in the process group. Each GPU contributes its shard, and everyone receives the complete, full-precision parameter tensor. Now you temporarily have 100% of the weights.

3. Compute the layer: With full parameters materialized in memory, the GPU performs the standard forward computation (matrix multiplications, activations, etc.) as if this were single-GPU training.

4. Discard unneeded shards: Immediately after computation completes, FSDP frees the parameter shards from other GPUs. Your device retains only its own 1/N shard, drastically reducing memory footprint.

5. Repeat per layer: This gather→compute→discard cycle repeats for each FSDP-wrapped layer as activations flow through the network.

The Memory Trade-off

Think of it like a library system: you don't own all books (parameters), but you can temporarily borrow the complete set when needed, then return what you don't own. The cost is communication time; the benefit is fitting models that would otherwise exceed GPU memory.

Key Takeaway: FSDP's forward pass uses all-gather to temporarily materialize full parameters for computation, then immediately discards non-owned shards—trading communication overhead for massive memory savings layer by layer.