Course contentsShow
← Machine Learning and Deep Learning
Lesson 2736 of 3,53859. Distributed Training: Data ParallelismFree lesson

Auto Wrap Policy in PyTorch FSDP

Using size-based and transformer-layer-based auto-wrap policies to shard models without manual configuration.

Auto Wrap Policy in PyTorch FSDP

What you'll learn: How to automatically shard large models in FSDP using size-based and transformer-layer-based wrapping policies without manually configuring each layer.

The Manual Wrapping Problem

In previous lessons, you learned about Unit vs Full Shard wrapping strategies—but manually wrapping every layer is tedious and error-prone, especially in massive models with hundreds of layers. Auto wrap policies solve this by automatically deciding which layers or modules should be wrapped as FSDP units.

Core Idea: Rule-Based Automatic Wrapping

An auto wrap policy is a set of rules that tells FSDP: "wrap modules that meet these criteria." PyTorch FSDP provides two main policies:

Size-Based Auto Wrap

This policy wraps any module whose parameter count exceeds a threshold. Think of it like packing boxes: if a module has more than N million parameters, it gets its own FSDP wrapper (its own "box").

from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy

# Wrap modules with >100M parameters
auto_wrap_policy = size_based_auto_wrap_policy(min_num_params=100_000_000)
model = FSDP(model, auto_wrap_policy=auto_wrap_policy)

Transformer-Layer-Based Auto Wrap

This policy targets specific layer types (like TransformerEncoderLayer). It's like saying "every transformer block gets its own wrapper"—ideal for models with repeated architectural patterns.

from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy
from torch.nn import TransformerEncoderLayer

auto_wrap_policy = transformer_auto_wrap_policy(
    transformer_layer_cls={TransformerEncoderLayer}
)
model = FSDP(model, auto_wrap_policy=auto_wrap_policy)

Why This Matters

Auto wrap policies save you from manually instrumenting every module while ensuring balanced memory distribution across GPUs. Size-based works for any architecture; transformer-based is optimized for the dominant model family today.

Key Takeaway: Auto wrap policies let FSDP automatically shard models using size thresholds or layer-type patterns, eliminating manual configuration for complex architectures.