Course contentsShow
← Machine Learning and Deep Learning
Lesson 1716 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Where to Apply LoRA: Target Modules

Selecting which layers to adapt: query, key, value projections, and feed-forward layers.

Where to Apply LoRA: Target Modules

What you'll learn: Which specific layers in a transformer should receive LoRA adapters for optimal efficiency and performance.

Understanding Target Module Selection

Now that you know LoRA adds low-rank updates to frozen weights, the practical question becomes: which weights should you adapt? A transformer model has dozens of weight matrices across its many layers—you don't need to (and shouldn't) apply LoRA everywhere.

The most common and effective targets are the attention mechanism's projection matrices:

  • Query (Q) projection: Transforms input into query vectors
  • Key (K) projection: Creates key vectors for attention scoring
  • Value (V) projection: Produces value vectors to be weighted
  • Output projection: Projects attention outputs back to model dimension

These are prime candidates because attention is where the model learns context-specific patterns—exactly what fine-tuning aims to adapt.

Feed-Forward Layers as Targets

The feed-forward networks (FFN) in each transformer block also contain large weight matrices. These can be targeted with LoRA, though they're often considered secondary priorities. FFN layers store factual knowledge and transformations, so adapting them can help with domain-specific tasks.

Common strategies:

  • Q+V only: Lightweight, often sufficient for many tasks
  • Q+K+V+Output: More comprehensive attention adaptation
  • All attention + FFN: Maximum flexibility, higher parameter count

Practical Trade-offs

Think of target module selection like choosing which controls to expose on a complex machine. More targets = more flexibility but also more trainable parameters. Starting with Q and V projections gives you adaptation power where it matters most—in how the model attends to information.

Key Takeaway: Query and Value projections are the most common LoRA targets because they directly control attention patterns; adding Key, Output, and feed-forward layers increases adaptation capacity at the cost of more parameters.