Course contentsShow
← Machine Learning and Deep Learning
Lesson 2743 of 3,53859. Distributed Training: Data ParallelismFree lesson

Memory Bottlenecks in Large Model Training

Understanding optimizer states, gradients, and parameters as sources of GPU memory consumption during training.

Memory Bottlenecks in Large Model Training

What you'll learn: You'll understand the three main sources of GPU memory consumption during large model training and why memory becomes the limiting factor before compute.

The Memory Problem

When training large models, you'll hit memory limits long before you exhaust compute capacity. But what exactly eats up all that GPU memory? The answer lies in three key components that must coexist during training.

The Three Memory Culprits

1. Model Parameters

These are your model weights—the actual numbers that define your network. A 7B parameter model in float32 requires roughly 28GB just for the weights themselves (4 bytes × 7 billion).

2. Gradients

For every parameter, backpropagation computes a gradient. This doubles your memory footprint: if parameters take 28GB, gradients take another 28GB.

3. Optimizer States

This is the silent killer. Optimizers like Adam maintain momentum and variance statistics for each parameter—that's two additional copies of your parameters. Now your 28GB parameter set balloons to 112GB total (parameters + gradients + 2× optimizer states).

Why This Matters

Think of GPU memory like a small apartment: you need space for your furniture (parameters), your renovation plans (gradients), and your moving history records (optimizer states). As models grow from millions to billions of parameters, this memory footprint becomes the primary bottleneck—not training speed.

This is precisely why techniques like FSDP (which shards these components across GPUs) and gradient accumulation (which reduces effective batch size per GPU) exist: they're workarounds for the memory wall.

Key Takeaway: Large model training is primarily memory-bound, with optimizer states typically consuming 50% of total memory, followed by parameters and gradients at ~25% each—understanding this breakdown is essential for choosing the right distributed training strategy.