LoRA Core Concept: Frozen Weights Plus Low-Rank Updates
What you'll learn: How LoRA avoids updating billions of pretrained parameters by keeping them frozen and learning small additive changes instead.
The Core Innovation
LoRA (Low-Rank Adaptation) solves the parameter efficiency problem with an elegant trick: instead of modifying the pretrained weight matrix W during fine-tuning, LoRA keeps W completely frozen and learns a separate low-rank update ΔW that gets added during inference.
For a weight matrix of dimension d × d, LoRA decomposes the update as:
ΔW = B × A
where B is d × r and A is r × d, with r (the rank) being much smaller than d (typically r = 8, 16, or 32).
Why This Works
Think of it like photo editing: you don't rewrite every pixel of your original image (the frozen pretrained weights). Instead, you add a thin adjustment layer on top (the low-rank update). The original stays pristine, but the output changes.
During forward passes:
output = W·x + B·A·x
The pretrained knowledge in W remains intact, while B and A (containing far fewer parameters) capture task-specific adaptations.
The Mathematics of Efficiency
If W has 1 million parameters (1000 × 1000), full fine-tuning updates all 1 million. With LoRA at rank r=16:
- A: 1000 × 16 = 16,000 parameters
- B: 16 × 1000 = 16,000 parameters
- Total trainable: 32,000 parameters (97% reduction!)
Key Takeaway: LoRA freezes pretrained weights entirely and learns task adaptations through tiny low-rank matrices that get added to the original weights, dramatically reducing memory and enabling efficient multi-task deployment.