Course contentsShow
← Machine Learning and Deep Learning
Lesson 1713 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

LoRA Core Concept: Frozen Weights Plus Low-Rank Updates

How LoRA keeps pretrained weights frozen and learns additive low-rank adaptations instead.

LoRA Core Concept: Frozen Weights Plus Low-Rank Updates

What you'll learn: How LoRA avoids updating billions of pretrained parameters by keeping them frozen and learning small additive changes instead.

The Core Innovation

LoRA (Low-Rank Adaptation) solves the parameter efficiency problem with an elegant trick: instead of modifying the pretrained weight matrix W during fine-tuning, LoRA keeps W completely frozen and learns a separate low-rank update ΔW that gets added during inference.

For a weight matrix of dimension d × d, LoRA decomposes the update as:

ΔW = B × A

where B is d × r and A is r × d, with r (the rank) being much smaller than d (typically r = 8, 16, or 32).

Why This Works

Think of it like photo editing: you don't rewrite every pixel of your original image (the frozen pretrained weights). Instead, you add a thin adjustment layer on top (the low-rank update). The original stays pristine, but the output changes.

During forward passes:

output = W·x + B·A·x

The pretrained knowledge in W remains intact, while B and A (containing far fewer parameters) capture task-specific adaptations.

The Mathematics of Efficiency

If W has 1 million parameters (1000 × 1000), full fine-tuning updates all 1 million. With LoRA at rank r=16:

  • A: 1000 × 16 = 16,000 parameters
  • B: 16 × 1000 = 16,000 parameters
  • Total trainable: 32,000 parameters (97% reduction!)

Key Takeaway: LoRA freezes pretrained weights entirely and learns task adaptations through tiny low-rank matrices that get added to the original weights, dramatically reducing memory and enabling efficient multi-task deployment.