Memory Requirements for Full Fine-Tuning
You'll learn: Why full fine-tuning demands vastly more GPU memory than inference alone, and what exactly fills that memory budget.
The Hidden Memory Multiplier
When you run inference on an LLM, you only need to store the model's weights in memory. But full fine-tuning? That's a different beast entirely. You're not just loading the model—you're updating every parameter through backpropagation, which requires storing multiple copies of data at each layer.
The Four Memory Components
1. Model Weights
The parameters themselves (your 7B model's ~28GB in FP32). This is your baseline.
2. Gradients
During backpropagation, you compute a gradient for every parameter. That's another full copy of your model's size—another ~28GB.
3. Optimizer States
Optimizers like Adam track momentum and variance for each parameter. Adam stores two states per parameter, effectively doubling your memory again—roughly ~56GB more.
4. Activations
These are the intermediate outputs stored during the forward pass, needed for backpropagation. Size depends on batch size and sequence length. For long contexts or large batches, activations can rival or exceed the model size itself.
The Math That Hurts
Rule of thumb: Full fine-tuning with Adam requires roughly 4× the model parameter memory (weights + gradients + 2 optimizer states), plus activation memory.
For a 7B parameter model in FP32:
- Weights: 28GB
- Gradients: 28GB
- Optimizer states: 56GB
- Activations: 10-50GB+ (varies)
Total: ~122GB+ for a "small" 7B model!
This is why full fine-tuning often requires multiple high-end GPUs or carefully engineered memory optimizations.
Key Takeaway: Full fine-tuning multiplies memory needs by ~4× (before activations) because you must store weights, gradients, and optimizer states simultaneously—making it impractical for large models on consumer hardware.