Low-Rank Adaptation (LoRA) Core Concept
What You'll Learn
You'll understand how LoRA achieves efficient fine-tuning by freezing the original model weights and training tiny rank decomposition matrices instead.
The Core Idea
Imagine you have a massive library catalog system (your pre-trained model) with millions of entries. Instead of rewriting the entire catalog for a new specialty collection, you create a small index card system that notes only the changes needed. That's LoRA.
LoRA (Low-Rank Adaptation) keeps all the original model weights completely frozen—you never touch them during fine-tuning. Instead, it adds pairs of small matrices (called rank decomposition matrices) alongside the existing weight matrices. These small matrices learn the adjustments needed for your specific task.
How It Works
When fine-tuning a traditional model, you might update millions or billions of parameters. LoRA instead adds two small matrices—let's call them A and B—where:
- A has dimensions
[original_dimension × rank] - B has dimensions
[rank × original_dimension] - The
rankis typically very small (8, 16, 32, or 64)
During inference, the output becomes: original_weights × input + (B × A) × input
The magic is that B × A approximates the weight updates you'd get from full fine-tuning, but with far fewer parameters to train. For a weight matrix of 1000×1000, full fine-tuning trains 1,000,000 parameters. With rank=16, LoRA trains only 32,000 parameters—a 97% reduction!
Why This Matters
You've already learned why PEFT is valuable. LoRA is the most popular PEFT technique because it maintains model quality while dramatically reducing memory, storage, and training costs.
Key Takeaway: LoRA freezes base model weights and trains small rank decomposition matrices that capture task-specific adaptations, reducing trainable parameters by 90-99% while maintaining fine-tuning effectiveness.