Course contentsShow
← Machine Learning and Deep Learning
Lesson 2737 of 3,53859. Distributed Training: Data ParallelismFree lesson

CPU Offloading in FSDP

Offloading parameters and gradients to CPU memory between computation steps to maximize model size at cost of speed.

CPU Offloading in FSDP

What you'll learn: How FSDP can offload parameters and gradients to CPU memory to train even larger models when GPU memory is exhausted.

The Core Idea

When training massive models with FSDP, even after sharding parameters across GPUs, you might still run out of GPU memory. CPU offloading extends your capacity by temporarily moving parameters, gradients, or optimizer states to CPU RAM between computation steps.

Think of it like a librarian with a small desk (GPU memory) but a huge archive room (CPU RAM). Instead of keeping all books on the desk, the librarian fetches only what's needed for the current task, then returns it to the archive.

How It Works in FSDP

During the forward pass, FSDP normally all-gathers sharded parameters onto GPU. With CPU offloading enabled:

  1. Parameters are stored on CPU by default
  2. Before computation, FSDP moves needed parameters to GPU
  3. Computation executes on GPU as usual
  4. After computation, parameters return to CPU
  5. Gradients are computed on GPU, then optionally offloaded to CPU
  6. Optimizer states can also live on CPU between updates

The all-gather and reduce-scatter operations still happen, but between CPU and GPU rather than staying GPU-resident.

The Trade-Off

Benefit: You can train models that simply won't fit in GPU memory otherwise—the main constraint becomes CPU RAM (typically 100s of GB) instead of GPU RAM (10-80 GB).

Cost: Substantial slowdown due to PCIe bandwidth limits. Moving data between CPU and GPU is much slower than GPU-to-GPU communication. Expect 2-5× longer training times.

Key Takeaway: CPU offloading in FSDP lets you maximize model size at the cost of training speed by using CPU memory as overflow storage for parameters and gradients between GPU computation steps.