Memory Benefits: Training Only a Fraction of Parameters
You'll learn how LoRA dramatically reduces memory requirements by freezing most model weights and training only small adapter matrices.
The Memory Problem in Traditional Fine-Tuning
When you fine-tune a large language model the traditional way, you update every parameter. This means you need to:
- Store gradients for every weight
- Keep optimizer states (momentum, variance) for every parameter
- Maintain activations for backpropagation through all layers
For a 7B parameter model, optimizer states alone can consume 56GB of memory with AdamW!
How LoRA Slashes Memory Usage
LoRA freezes the original pretrained weights completely. No gradients are computed for frozen parameters. Instead, you train only the small low-rank matrices (A and B) that you've added.
If you apply LoRA with rank r=8 to the attention layers (about 25% of parameters in typical models), you might train only 0.1-0.5% of the total parameters.
The Memory Savings Breakdown
Without LoRA (7B model):
- Model weights: 14GB (float16)
- Gradients: 14GB
- Optimizer states: 56GB
- Total: ~84GB
With LoRA (r=8, selective modules):
- Frozen weights: 14GB (no gradients!)
- LoRA parameters: ~8-35MB
- Gradients for LoRA: ~8-35MB
- Optimizer states for LoRA: ~32-140MB
- Total: ~14.2GB
You've just reduced memory by 80-85% while maintaining comparable performance!
Key Takeaway: LoRA's memory efficiency comes from freezing the vast majority of parameters—no gradients or optimizer states needed for them—while training only tiny low-rank adapters that capture the essential task-specific adjustments.