Course contentsShow
← AI Engineering
Lesson 1350 of 1,88633. Parameter-Efficient Fine-Tuning (PEFT)Free lesson

Target Modules and Layer Selection

Choosing which model layers to apply LoRA adapters to: query, key, value, and output projections.

Target Modules and Layer Selection

What you'll learn: How to strategically choose which transformer layers receive LoRA adapters to maximize efficiency while maintaining model quality.

Core Concept

When applying LoRA, you don't have to adapt every layer in your transformer model. In fact, selectively targeting specific projection matrices often yields better results with fewer trainable parameters. The typical candidates are:

  • Query (Q) projections – Controls what the attention mechanism "looks for"
  • Key (K) projections – Controls what the attention mechanism "matches against"
  • Value (V) projections – Controls what information flows through
  • Output projections – Controls the final attention output transformation

Think of it like renovating a house: you don't need to tear down every wall. Sometimes updating just the kitchen and bathroom gives you 90% of the value at 30% of the cost.

Which Layers to Target?

Most practitioners start with query and value projections (q_proj, v_proj) as they tend to provide the best quality-to-parameter ratio. For more complex tasks, you might add key and output projections.

The choice depends on your task:

  • Simple style/format changes: Q and V often suffice
  • Domain adaptation: Add K and output projections
  • Complex reasoning: Consider all attention projections

You can also target different depths: early layers capture general features, while deeper layers encode task-specific patterns. Targeting only the final 25-50% of layers can reduce parameters while preserving performance.

Practical Considerations

Fewer target modules mean faster training and inference, but may limit expressiveness. Start conservative (Q+V only), measure performance, then expand if needed.

Key Takeaway: Target modules strategically—you rarely need to adapt every layer. Query and value projections typically offer the best efficiency-quality balance, though your specific task may benefit from broader or narrower targeting.