Prefix Tuning: Prepending Learnable Vectors
What you'll learn: How prefix tuning adapts LLMs by learning task-specific vectors that are prepended to the attention mechanism's keys and values, rather than modifying model weights.
The Core Idea
Instead of adding adapter layers between transformer blocks or training low-rank weight updates like LoRA, prefix tuning takes a different approach: it prepends learnable "prefix" vectors to the keys and values in each attention layer. These prefixes act like virtual tokens that guide the model's behavior for specific tasks without changing the original model weights at all.
Think of it like adding a set of persistent instructions at the beginning of every conversation. Rather than retraining how someone thinks (the model weights), you're giving them a reference card they consult for every decision.
How It Works
In standard attention, queries interact with keys and values. Prefix tuning adds learned prefix vectors to the keys and values at each transformer layer:
- Original:
Attention(Q, K, V) - With Prefix:
Attention(Q, [P_k; K], [P_v; V])
Here, P_k and P_v are learnable prefix vectors concatenated with the original keys and values. Only these prefix parameters are updated during training—the entire transformer stays frozen.
The prefix length (typically 10-200 tokens worth) is a key hyperparameter. Longer prefixes offer more capacity but require more memory and computation.
Advantages
- Minimal parameters: Only the prefix vectors are trainable
- No architecture changes: Works with any existing transformer
- Task switching: Different prefixes for different tasks, easily swappable at inference
Key Takeaway: Prefix tuning optimizes task-specific vectors prepended to each layer's attention mechanism, guiding model behavior without modifying weights—offering parameter efficiency similar to adapters but with a fundamentally different architectural approach.