Course contentsShow
← Machine Learning and Deep Learning
Lesson 2735 of 3,53859. Distributed Training: Data ParallelismFree lesson

Unit vs Full Shard Wrapping Strategies

Choosing granularity for FSDP wrapping: per-layer, per-block, or full model and their memory-communication tradeoffs.

Unit vs Full Shard Wrapping Strategies

You'll learn: How to choose the right granularity when wrapping model components with FSDP to balance memory savings against communication overhead.

The Core Trade-off

When using Fully Sharded Data Parallel (FSDP), you must decide what to wrap: individual layers, logical blocks (like Transformer layers), or the entire model as one unit. This choice dramatically affects both memory footprint and communication efficiency.

Think of it like packing for a move: You can wrap each small item individually (fine-grained), pack related items into boxes (medium-grained), or shrink-wrap entire furniture sets together (coarse-grained). Each approach has different space and handling costs.

Wrapping Granularity Options

Unit/Layer-Level Wrapping: Wrap each individual layer (e.g., every Linear or Conv2d). This maximizes memory savings because parameters are sharded at the finest granularity, staying sharded except during that specific layer's computation.

Block-Level Wrapping: Wrap logical modules (e.g., entire Transformer blocks containing attention + FFN). Balances memory and communication—parameters for the whole block are gathered once, used for multiple operations, then discarded.

Full Model Wrapping: Wrap the entire model as a single FSDP unit. Minimal communication overhead but reduced memory benefits, since all parameters must be gathered at once.

Memory-Communication Trade-off

Finer wrapping (per-layer) = more all-gather operations (higher communication) but lower peak memory (parameters released immediately after each layer).

Coarser wrapping (full model) = fewer all-gather operations (lower communication) but higher peak memory (all parameters present simultaneously).

The sweet spot for large models like Transformers is typically block-level wrapping—it exploits natural computational boundaries, amortizing communication costs while keeping memory manageable.

Key Takeaway: Choose FSDP wrapping granularity based on your model's structure and constraints—fine-grained for maximum memory savings, coarse-grained for speed, or block-level for the best balance in hierarchical architectures.