LoRA Mathematics: Decomposing Weight Updates
You'll learn: How LoRA mathematically represents weight updates as a sum of frozen original weights plus a low-rank decomposition.
The Mathematical Heart of LoRA
In full fine-tuning, we update a weight matrix W to W' by applying gradients. LoRA takes a different approach: it keeps W frozen and instead learns the change (ΔW) as a product of two smaller matrices.
The key formula is:
W' = W + BA
Where:
- W is the original pretrained weight matrix (frozen, never updated)
- B is a matrix of size
(d × r) - A is a matrix of size
(r × k) - r is the rank (typically 8, 16, or 64 — much smaller than d or k)
Why This Works
Think of W as a detailed map of knowledge. Instead of redrawing the entire map, LoRA adds a transparent overlay (BA) with just the new information. The magic is that BA has far fewer parameters than W because r is small.
If W is 4096×4096 (16M parameters), but r=16, then:
- B has 4096×16 = 65K parameters
- A has 16×4096 = 65K parameters
- Total: 130K parameters instead of 16M (a 99%+ reduction!)
During inference: output = (W + BA) × input
The low-rank constraint means we're assuming the fine-tuning update lives in a much lower-dimensional space than the full weight matrix — which empirically proves true for most adaptation tasks.
Key Takeaway: LoRA decomposes weight updates as W + BA where the product BA captures task-specific adaptations with dramatically fewer parameters than updating W directly, exploiting the low intrinsic dimensionality of fine-tuning changes.