Course contentsShow
← Machine Learning and Deep Learning
Lesson 2750 of 3,53859. Distributed Training: Data ParallelismFree lesson

ZeRO-Infinity: NVMe Offloading

Extending offloading to NVMe storage for training models that exceed CPU and GPU memory combined.

ZeRO-Infinity: NVMe Offloading

What You'll Learn

You'll learn how ZeRO-Infinity extends memory offloading to NVMe storage, enabling training of models that exceed the combined capacity of GPU and CPU memory.

The Memory Hierarchy Extension

You've already seen ZeRO-Offload move optimizer states and gradients to CPU RAM when GPU memory runs out. But what happens when your model is so large that even CPU memory isn't enough?

ZeRO-Infinity adds another tier to the memory hierarchy: NVMe storage (think: fast SSDs). This creates a three-level system:

  • GPU memory (fastest, smallest)
  • CPU memory (medium speed, medium capacity)
  • NVMe storage (slower, massive capacity)

How It Works

Think of it like organizing your workspace: frequently-used tools stay on your desk (GPU), occasionally-needed items go in drawers (CPU), and rarely-accessed supplies live in storage closets (NVMe).

ZeRO-Infinity dynamically moves parameters, gradients, and optimizer states between these three tiers based on when they're needed. During the forward pass, parameters are pulled from NVMe → CPU → GPU just before computation. After the backward pass, gradients and updated states can be pushed back down the hierarchy.

The key insight: bandwidth management. Modern NVMe drives offer 3-7 GB/s throughput. By overlapping computation with data transfers and using efficient scheduling, ZeRO-Infinity keeps GPUs busy while hiding most of the latency from disk access.

This enables training models with trillions of parameters on modest GPU clusters—something previously impossible without specialized supercomputing infrastructure.

Key Takeaway: ZeRO-Infinity extends the memory hierarchy to NVMe storage, enabling training of models that exceed GPU+CPU memory by intelligently managing data movement across three tiers while overlapping transfers with computation.