Prompt Tuning vs Prefix Tuning
You'll learn: How prompt tuning and prefix tuning differ in where they inject learnable parameters and how that affects their effectiveness.
Core Idea
Both prompt tuning and prefix tuning add learnable "soft" parameters to adapt a frozen LLM, but they differ fundamentally in where those parameters live:
-
Prompt tuning prepends learnable token embeddings only at the input layer — you're essentially learning a continuous prompt that gets processed like regular text through all transformer layers.
-
Prefix tuning inserts learnable key-value pairs at every transformer layer — these prefix vectors directly influence the attention mechanism at each level of the network.
The Analogy
Think of a building with multiple floors (transformer layers):
-
Prompt tuning is like giving special instructions to visitors at the lobby entrance. Those instructions travel with visitors through all floors, but each floor processes them normally.
-
Prefix tuning is like stationing a helpful guide on every floor who can directly assist visitors at each level. More intervention points means potentially more control.
Effectiveness Comparison
Prefix tuning generally outperforms prompt tuning, especially for smaller models, because:
- It influences representations at all layers, not just the input
- Each layer's prefix can capture task-specific information appropriate to that level of abstraction
However, prefix tuning requires storing more parameters (prefix vectors × number of layers), while prompt tuning is simpler and uses fewer parameters.
For very large models (100B+ parameters), the gap narrows — prompt tuning can approach prefix tuning's performance with sufficient model capacity.
Key Takeaway: Prompt tuning adds soft prompts only at the input, while prefix tuning adds learnable prefixes at every layer; prefix tuning typically performs better but uses more parameters.