Course contentsShow
← AI Engineering
Lesson 1359 of 1,88633. Parameter-Efficient Fine-Tuning (PEFT)Free lesson

Understanding LoRA Adapters and Model Architecture

Learn how LoRA adapters modify base models without changing original weights and how they integrate with transformer layers.

Understanding LoRA Adapters and Model Architecture

You'll learn how LoRA adapters plug into transformer layers without altering the original model weights, creating a modular fine-tuning architecture.

How LoRA Modifies Models Non-Destructively

When you apply LoRA to a base model, you're not rewriting its parameters. Instead, LoRA injects small adapter matrices alongside the original weights. Think of it like adding a translucent overlay to a painting—the original artwork remains untouched beneath, but the final image incorporates both layers.

The base model's weight matrix W stays frozen. LoRA adds two small matrices A and B whose product approximates the weight update: W' = W + BA. During forward passes, the model computes output = W·input + BA·input, combining frozen and trainable paths.

Integration with Transformer Layers

LoRA adapters typically target attention mechanism matrices: query (q_proj), key (k_proj), value (v_proj), and output (o_proj) projections. You can also apply them to feed-forward layers, though attention layers usually yield the best results.

Each targeted layer gets its own pair of low-rank matrices. The adapter sits in parallel with the original projection:

  • Original path: frozen weights process the input
  • LoRA path: B·A·input adds the learned adaptation
  • Both paths sum before the activation function

This architectural choice means you can swap adapters at inference time without touching the base model. Load different LoRA weights for different tasks, and the same base model serves multiple purposes with minimal memory overhead.

Why This Matters for Deployment

Because adapters are separate and tiny (often <1% of base model size), you can store dozens of specialized versions and hot-swap them based on user needs—all while keeping one copy of the base model in memory.

Key Takeaway: LoRA adapters modify model behavior by injecting trainable low-rank matrices in parallel with frozen transformer layers, enabling non-destructive, swappable fine-tuning without changing original weights.