Course contentsShow
← Machine Learning and Deep Learning
Lesson 2729 of 3,53859. Distributed Training: Data ParallelismFree lesson

FSDP Motivation: Beyond DDP Memory Limits

Why DDP replicates full model copies per GPU and how FSDP enables training larger models by sharding parameters.

FSDP Motivation: Beyond DDP Memory Limits

What you'll learn: Why standard DDP breaks down for massive models and how FSDP enables training networks far too large to fit on a single GPU.

The DDP Memory Wall

In Distributed Data Parallel (DDP), each GPU maintains a complete copy of the entire model—all parameters, gradients, and optimizer states. This works beautifully for models up to a few billion parameters, but modern language models like GPT-3 (175B parameters) or larger simply won't fit in a single GPU's memory, even if you use just one worker.

Think of it like photocopying a massive encyclopedia: if every person in your study group needs their own full copy, you'll run out of shelf space quickly, no matter how many people you have.

FSDP's Solution: Shard Everything

Fully Sharded Data Parallel (FSDP) breaks this pattern by sharding (partitioning) the model's parameters, gradients, and optimizer states across all GPUs. Each GPU only stores a fraction of the model. During forward and backward passes, FSDP dynamically gathers the necessary parameters from other GPUs on-demand, then discards them after use.

Instead of everyone holding the full encyclopedia, each person holds just one volume. When you need Chapter 15, you temporarily borrow it from your neighbor, read what you need, and return it.

This dramatically reduces per-GPU memory consumption, enabling you to train models that would otherwise require hundreds of GPUs—or be impossible entirely—using just a handful of devices.

Key Takeaway: DDP replicates full model copies per GPU, limiting maximum model size to what fits in one device's memory; FSDP shards parameters across GPUs, enabling training of models far larger than any single GPU can hold.