Course contentsShow
← AI Engineering
Lesson 1348 of 1,88633. Parameter-Efficient Fine-Tuning (PEFT)Free lesson

Low-Rank Adaptation (LoRA) Core Concept

How LoRA freezes base weights and trains small rank decomposition matrices to achieve efficient fine-tuning.

Low-Rank Adaptation (LoRA) Core Concept

What You'll Learn

You'll understand how LoRA achieves efficient fine-tuning by freezing the original model weights and training tiny rank decomposition matrices instead.

The Core Idea

Imagine you have a massive library catalog system (your pre-trained model) with millions of entries. Instead of rewriting the entire catalog for a new specialty collection, you create a small index card system that notes only the changes needed. That's LoRA.

LoRA (Low-Rank Adaptation) keeps all the original model weights completely frozen—you never touch them during fine-tuning. Instead, it adds pairs of small matrices (called rank decomposition matrices) alongside the existing weight matrices. These small matrices learn the adjustments needed for your specific task.

How It Works

When fine-tuning a traditional model, you might update millions or billions of parameters. LoRA instead adds two small matrices—let's call them A and B—where:

  • A has dimensions [original_dimension × rank]
  • B has dimensions [rank × original_dimension]
  • The rank is typically very small (8, 16, 32, or 64)

During inference, the output becomes: original_weights × input + (B × A) × input

The magic is that B × A approximates the weight updates you'd get from full fine-tuning, but with far fewer parameters to train. For a weight matrix of 1000×1000, full fine-tuning trains 1,000,000 parameters. With rank=16, LoRA trains only 32,000 parameters—a 97% reduction!

Why This Matters

You've already learned why PEFT is valuable. LoRA is the most popular PEFT technique because it maintains model quality while dramatically reducing memory, storage, and training costs.

Key Takeaway: LoRA freezes base model weights and trains small rank decomposition matrices that capture task-specific adaptations, reducing trainable parameters by 90-99% while maintaining fine-tuning effectiveness.