Memory Bottlenecks in Full Fine-Tuning
What you'll learn: You'll understand exactly why fine-tuning large language models consumes so much GPU memory and where that memory goes.
The Four Memory Consumers
When you fine-tune an LLM, GPU memory doesn't just store the model weights—it needs to track four major components simultaneously:
1. Model Weights
The baseline parameters of your LLM (e.g., 7B parameters × 2 bytes for FP16 = 14GB just for weights).
2. Optimizer States
Optimizers like Adam track momentum and variance for every parameter. Adam requires 2 additional copies of all parameters (8 bytes per parameter total), tripling your memory footprint before you even start training.
3. Gradients
During backpropagation, you compute gradients for every parameter. That's another full copy of the model size (2 bytes per parameter in FP16).
4. Activations
Intermediate outputs from each layer during the forward pass must be kept in memory for backpropagation. For large batch sizes or long sequences, activations can be the largest memory consumer, sometimes exceeding the model size itself.
The Memory Math
For a 7B parameter model with Adam optimizer:
- Weights: 14GB
- Optimizer states: 28GB
- Gradients: 14GB
- Activations: 10-40GB (varies with batch size and sequence length)
Total: 66-96GB of memory needed—far exceeding most consumer GPUs.
This is why full fine-tuning remains impractical for most researchers without expensive multi-GPU setups, and why parameter-efficient methods like LoRA have become essential.
Key Takeaway: Full fine-tuning requires roughly 12-16× the memory of just loading the model weights due to optimizer states, gradients, and activations—making it the primary bottleneck for training large models.