Mixed Precision with FSDP
What you'll learn: How to configure FSDP to compute in bfloat16 or float16 while keeping full precision master weights in the sharded optimizer state.
The Core Idea
When you combine Fully Sharded Data Parallel (FSDP) with mixed precision training, you get the memory savings of both techniques stacked together. FSDP already shards model parameters and gradients across GPUs. Mixed precision further reduces memory by using 16-bit data types (bfloat16 or float16) during forward and backward passes—but here's the key twist: the optimizer's master weights remain in float32 precision, and these too are sharded.
How It Works in FSDP
Think of it like a library where books (parameters) are split across multiple shelves (GPUs), but each shelf also maintains a master catalog (float32 master weights) that's more detailed than the public-facing copies (float16/bfloat16 compute weights).
During the forward pass, FSDP all-gathers the float16/bfloat16 shards just-in-time for computation. During the backward pass, gradients are computed in reduced precision, then reduced-scattered back to each rank's shard. But when the optimizer updates weights, it works with the float32 master copy stored in its sharded state, then casts back down to float16/bfloat16 for the next iteration.
This approach prevents precision loss from accumulating over many training steps while maintaining memory efficiency.
Configuration
In PyTorch FSDP, you specify mixed precision via a MixedPrecision policy that defines:
param_dtype: the precision for parameters (e.g.,torch.bfloat16)reduce_dtype: the precision for gradient reduction (often matches param_dtype)buffer_dtype: precision for buffers like batch norm statistics
The optimizer internally maintains float32 master weights in its sharded state.
Key Takeaway: FSDP with mixed precision computes in float16/bfloat16 for speed and memory savings while maintaining float32 master weights in the sharded optimizer, combining the benefits of both FSDP and mixed precision without sacrificing numerical stability.