FSDP vs DDP: When to Use Each
You'll learn: How to choose between Fully Sharded Data Parallel (FSDP) and Distributed Data Parallel (DDP) based on model characteristics and hardware constraints.
The Decision Framework
Think of DDP and FSDP as two tools for different jobs: DDP is like having each worker carry a complete toolkit, while FSDP distributes pieces of the toolkit across workers who share as needed. Your choice depends on four key factors.
Model Size vs GPU Memory
Use DDP when: Your model comfortably fits in a single GPU's memory with room for gradients and optimizer states. For models under 1-2B parameters on modern GPUs, DDP is typically simpler and faster.
Use FSDP when: Your model is too large to fit on one GPU. If you're running into OOM errors even with batch size 1, FSDP's memory sharding becomes essential. Models beyond 3B parameters almost always need FSDP.
Batch Size Considerations
DDP thrives with larger per-GPU batch sizes because its communication overhead is fixed per step—more computation per communication event improves efficiency.
FSDP allows training when you're forced into tiny batch sizes by model size. The gradient accumulation techniques you've learned work seamlessly with FSDP's sharding.
Communication Overhead Trade-offs
DDP only synchronizes gradients once per backward pass—minimal communication with maximum overlap potential.
FSDP performs all-gather and reduce-scatter operations throughout forward and backward passes. This increases communication frequency but enables training otherwise impossible models.
Key Takeaway: Use DDP for models that fit in GPU memory to minimize communication overhead; switch to FSDP when model size, not batch size, becomes your limiting factor—accepting more communication in exchange for memory efficiency.