Unit vs Full Shard Wrapping Strategies
You'll learn: How to choose the right granularity when wrapping model components with FSDP to balance memory savings against communication overhead.
The Core Trade-off
When using Fully Sharded Data Parallel (FSDP), you must decide what to wrap: individual layers, logical blocks (like Transformer layers), or the entire model as one unit. This choice dramatically affects both memory footprint and communication efficiency.
Think of it like packing for a move: You can wrap each small item individually (fine-grained), pack related items into boxes (medium-grained), or shrink-wrap entire furniture sets together (coarse-grained). Each approach has different space and handling costs.
Wrapping Granularity Options
Unit/Layer-Level Wrapping: Wrap each individual layer (e.g., every Linear or Conv2d). This maximizes memory savings because parameters are sharded at the finest granularity, staying sharded except during that specific layer's computation.
Block-Level Wrapping: Wrap logical modules (e.g., entire Transformer blocks containing attention + FFN). Balances memory and communication—parameters for the whole block are gathered once, used for multiple operations, then discarded.
Full Model Wrapping: Wrap the entire model as a single FSDP unit. Minimal communication overhead but reduced memory benefits, since all parameters must be gathered at once.
Memory-Communication Trade-off
Finer wrapping (per-layer) = more all-gather operations (higher communication) but lower peak memory (parameters released immediately after each layer).
Coarser wrapping (full model) = fewer all-gather operations (lower communication) but higher peak memory (all parameters present simultaneously).
The sweet spot for large models like Transformers is typically block-level wrapping—it exploits natural computational boundaries, amortizing communication costs while keeping memory manageable.
Key Takeaway: Choose FSDP wrapping granularity based on your model's structure and constraints—fine-grained for maximum memory savings, coarse-grained for speed, or block-level for the best balance in hierarchical architectures.