CPU Offloading in FSDP
What you'll learn: How FSDP can offload parameters and gradients to CPU memory to train even larger models when GPU memory is exhausted.
The Core Idea
When training massive models with FSDP, even after sharding parameters across GPUs, you might still run out of GPU memory. CPU offloading extends your capacity by temporarily moving parameters, gradients, or optimizer states to CPU RAM between computation steps.
Think of it like a librarian with a small desk (GPU memory) but a huge archive room (CPU RAM). Instead of keeping all books on the desk, the librarian fetches only what's needed for the current task, then returns it to the archive.
How It Works in FSDP
During the forward pass, FSDP normally all-gathers sharded parameters onto GPU. With CPU offloading enabled:
- Parameters are stored on CPU by default
- Before computation, FSDP moves needed parameters to GPU
- Computation executes on GPU as usual
- After computation, parameters return to CPU
- Gradients are computed on GPU, then optionally offloaded to CPU
- Optimizer states can also live on CPU between updates
The all-gather and reduce-scatter operations still happen, but between CPU and GPU rather than staying GPU-resident.
The Trade-Off
Benefit: You can train models that simply won't fit in GPU memory otherwise—the main constraint becomes CPU RAM (typically 100s of GB) instead of GPU RAM (10-80 GB).
Cost: Substantial slowdown due to PCIe bandwidth limits. Moving data between CPU and GPU is much slower than GPU-to-GPU communication. Expect 2-5× longer training times.
Key Takeaway: CPU offloading in FSDP lets you maximize model size at the cost of training speed by using CPU memory as overflow storage for parameters and gradients between GPU computation steps.