Auto Wrap Policy in PyTorch FSDP
What you'll learn: How to automatically shard large models in FSDP using size-based and transformer-layer-based wrapping policies without manually configuring each layer.
The Manual Wrapping Problem
In previous lessons, you learned about Unit vs Full Shard wrapping strategies—but manually wrapping every layer is tedious and error-prone, especially in massive models with hundreds of layers. Auto wrap policies solve this by automatically deciding which layers or modules should be wrapped as FSDP units.
Core Idea: Rule-Based Automatic Wrapping
An auto wrap policy is a set of rules that tells FSDP: "wrap modules that meet these criteria." PyTorch FSDP provides two main policies:
Size-Based Auto Wrap
This policy wraps any module whose parameter count exceeds a threshold. Think of it like packing boxes: if a module has more than N million parameters, it gets its own FSDP wrapper (its own "box").
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy
# Wrap modules with >100M parameters
auto_wrap_policy = size_based_auto_wrap_policy(min_num_params=100_000_000)
model = FSDP(model, auto_wrap_policy=auto_wrap_policy)
Transformer-Layer-Based Auto Wrap
This policy targets specific layer types (like TransformerEncoderLayer). It's like saying "every transformer block gets its own wrapper"—ideal for models with repeated architectural patterns.
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
from torch.nn import TransformerEncoderLayer
auto_wrap_policy = transformer_auto_wrap_policy(
transformer_layer_cls={TransformerEncoderLayer}
)
model = FSDP(model, auto_wrap_policy=auto_wrap_policy)
Why This Matters
Auto wrap policies save you from manually instrumenting every module while ensuring balanced memory distribution across GPUs. Size-based works for any architecture; transformer-based is optimized for the dominant model family today.
Key Takeaway: Auto wrap policies let FSDP automatically shard models using size thresholds or layer-type patterns, eliminating manual configuration for complex architectures.