Course contentsShow
← Machine Learning and Deep Learning
Lesson 2746 of 3,53859. Distributed Training: Data ParallelismFree lesson

ZeRO Stage 3: Parameter Partitioning

Full parameter sharding where each GPU holds only a slice of model weights, enabling massive models.

ZeRO Stage 3: Parameter Partitioning

You'll learn how ZeRO Stage 3 enables training models so large that their parameters won't fit on any single GPU by sharding the weights themselves.

The Core Idea

In ZeRO Stage 1 and 2, you partitioned optimizer states and gradients across GPUs, but every GPU still held a full copy of all model parameters. This creates a hard memory ceiling: if your model has 175 billion parameters, every GPU needs enough memory for all 175 billion weights.

ZeRO Stage 3 breaks this barrier by partitioning the model parameters themselves. Each GPU owns only a fraction of the weights. When a layer needs to compute, the required parameters are temporarily gathered via all-gather operations, used for the forward or backward pass, then immediately discarded. Only the local shard stays resident in memory.

How It Works

Think of it like a library with a massive encyclopedia set split across multiple shelves. When you need Volume 5, you temporarily borrow it from Shelf 5, read what you need, then return it. No single shelf holds all volumes permanently—only the one it's responsible for storing.

During forward pass: all-gather parameters → compute → discard non-local parameters
During backward pass: all-gather parameters → compute gradients → discard parameters → reduce-scatter gradients

This dramatically reduces per-GPU memory from O(model_size) to O(model_size / num_gpus).

Trade-offs

More communication overhead than Stage 1 or 2, since parameters fly across the network frequently. However, this unlocks training models that would otherwise be impossible.

Key Takeaway: ZeRO Stage 3 shards model parameters across GPUs, allowing you to train models far larger than any single device's memory by dynamically gathering and discarding weights as needed.