Course contentsShow
← Machine Learning and Deep Learning
Lesson 1740 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Prompt Tuning vs Prefix Tuning

Distinguish between soft prompt tuning at the input and prefix tuning at all layers, comparing their effectiveness.

Prompt Tuning vs Prefix Tuning

You'll learn: How prompt tuning and prefix tuning differ in where they inject learnable parameters and how that affects their effectiveness.

Core Idea

Both prompt tuning and prefix tuning add learnable "soft" parameters to adapt a frozen LLM, but they differ fundamentally in where those parameters live:

  • Prompt tuning prepends learnable token embeddings only at the input layer — you're essentially learning a continuous prompt that gets processed like regular text through all transformer layers.

  • Prefix tuning inserts learnable key-value pairs at every transformer layer — these prefix vectors directly influence the attention mechanism at each level of the network.

The Analogy

Think of a building with multiple floors (transformer layers):

  • Prompt tuning is like giving special instructions to visitors at the lobby entrance. Those instructions travel with visitors through all floors, but each floor processes them normally.

  • Prefix tuning is like stationing a helpful guide on every floor who can directly assist visitors at each level. More intervention points means potentially more control.

Effectiveness Comparison

Prefix tuning generally outperforms prompt tuning, especially for smaller models, because:

  • It influences representations at all layers, not just the input
  • Each layer's prefix can capture task-specific information appropriate to that level of abstraction

However, prefix tuning requires storing more parameters (prefix vectors × number of layers), while prompt tuning is simpler and uses fewer parameters.

For very large models (100B+ parameters), the gap narrows — prompt tuning can approach prefix tuning's performance with sufficient model capacity.

Key Takeaway: Prompt tuning adds soft prompts only at the input, while prefix tuning adds learnable prefixes at every layer; prefix tuning typically performs better but uses more parameters.