Course contentsShow
← Machine Learning and Deep Learning
Lesson 1714 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

LoRA Mathematics: Decomposing Weight Updates

The mathematical formulation of LoRA as W + BA where B and A are low-rank matrices.

LoRA Mathematics: Decomposing Weight Updates

You'll learn: How LoRA mathematically represents weight updates as a sum of frozen original weights plus a low-rank decomposition.

The Mathematical Heart of LoRA

In full fine-tuning, we update a weight matrix W to W' by applying gradients. LoRA takes a different approach: it keeps W frozen and instead learns the change (ΔW) as a product of two smaller matrices.

The key formula is:

W' = W + BA

Where:

  • W is the original pretrained weight matrix (frozen, never updated)
  • B is a matrix of size (d × r)
  • A is a matrix of size (r × k)
  • r is the rank (typically 8, 16, or 64 — much smaller than d or k)

Why This Works

Think of W as a detailed map of knowledge. Instead of redrawing the entire map, LoRA adds a transparent overlay (BA) with just the new information. The magic is that BA has far fewer parameters than W because r is small.

If W is 4096×4096 (16M parameters), but r=16, then:

  • B has 4096×16 = 65K parameters
  • A has 16×4096 = 65K parameters
  • Total: 130K parameters instead of 16M (a 99%+ reduction!)

During inference: output = (W + BA) × input

The low-rank constraint means we're assuming the fine-tuning update lives in a much lower-dimensional space than the full weight matrix — which empirically proves true for most adaptation tasks.

Key Takeaway: LoRA decomposes weight updates as W + BA where the product BA captures task-specific adaptations with dramatically fewer parameters than updating W directly, exploiting the low intrinsic dimensionality of fine-tuning changes.