Course contentsShow
← Machine Learning and Deep Learning
Lesson 1705 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Memory Requirements for Full Fine-Tuning

Understanding memory needed for model weights, gradients, optimizer states, and activations during full fine-tuning.

Memory Requirements for Full Fine-Tuning

You'll learn: Why full fine-tuning demands vastly more GPU memory than inference alone, and what exactly fills that memory budget.

The Hidden Memory Multiplier

When you run inference on an LLM, you only need to store the model's weights in memory. But full fine-tuning? That's a different beast entirely. You're not just loading the model—you're updating every parameter through backpropagation, which requires storing multiple copies of data at each layer.

The Four Memory Components

1. Model Weights

The parameters themselves (your 7B model's ~28GB in FP32). This is your baseline.

2. Gradients

During backpropagation, you compute a gradient for every parameter. That's another full copy of your model's size—another ~28GB.

3. Optimizer States

Optimizers like Adam track momentum and variance for each parameter. Adam stores two states per parameter, effectively doubling your memory again—roughly ~56GB more.

4. Activations

These are the intermediate outputs stored during the forward pass, needed for backpropagation. Size depends on batch size and sequence length. For long contexts or large batches, activations can rival or exceed the model size itself.

The Math That Hurts

Rule of thumb: Full fine-tuning with Adam requires roughly 4× the model parameter memory (weights + gradients + 2 optimizer states), plus activation memory.

For a 7B parameter model in FP32:

  • Weights: 28GB
  • Gradients: 28GB
  • Optimizer states: 56GB
  • Activations: 10-50GB+ (varies)

Total: ~122GB+ for a "small" 7B model!

This is why full fine-tuning often requires multiple high-end GPUs or carefully engineered memory optimizations.

Key Takeaway: Full fine-tuning multiplies memory needs by ~4× (before activations) because you must store weights, gradients, and optimizer states simultaneously—making it impractical for large models on consumer hardware.