Course contentsShow
← Machine Learning and Deep Learning
Lesson 2749 of 3,53859. Distributed Training: Data ParallelismFree lesson

ZeRO-Offload: CPU Memory Extension

Offloading optimizer states and gradients to CPU memory to train larger models on limited GPU resources.

ZeRO-Offload: CPU Memory Extension

What You'll Learn

You'll learn how ZeRO-Offload leverages CPU memory to train massive models that would otherwise exceed your GPU's memory capacity.

The Core Problem

Even with ZeRO's optimizer state and gradient partitioning (Stages 1 and 2), extremely large models can still exhaust GPU memory. ZeRO-Offload solves this by strategically moving specific training components to CPU RAM—which is typically far more abundant than GPU VRAM.

How ZeRO-Offload Works

The key insight: not all data needs GPU speed at all times.

ZeRO-Offload intelligently offloads optimizer states and gradients to CPU memory during training. Here's the workflow:

  1. Forward pass: Parameters stay on GPU (needed for fast computation)
  2. Backward pass: Gradients computed on GPU, then offloaded to CPU
  3. Optimizer step: Performed on CPU using offloaded optimizer states and gradients
  4. Parameter update: Updated parameters copied back to GPU for next iteration

Analogy: Think of your GPU as a chef's cutting board (fast but small) and CPU memory as a large pantry. You only keep ingredients on the cutting board when actively preparing them; everything else waits in the pantry until needed.

The Tradeoff

You're trading GPU memory for CPU-GPU transfer time. The optimizer step becomes slower (CPU compute is less parallel than GPU), and data movement adds overhead. However, this cost is often acceptable when it means the difference between training a model or running out of memory entirely.

ZeRO-Offload particularly shines when combined with ZeRO Stage 2, allowing you to train models 10x larger than would fit in GPU-only configurations.

Key Takeaway: ZeRO-Offload extends trainable model size by offloading optimizer states and gradients to abundant CPU memory, accepting communication overhead to overcome GPU memory limitations.