Prompt Lookup Decoding
What you'll learn: How to exploit n-gram overlap between input prompts and expected outputs to accelerate LLM inference without requiring a separate draft model.
The Core Idea
Prompt Lookup Decoding is a clever variant of speculative decoding that uses the prompt itself as a source of draft tokens. Instead of maintaining a separate draft model, it searches for n-gram matches between recently generated tokens and segments of the original input prompt.
This technique shines in scenarios with high input-output overlap—think code completion, document editing, or chat contexts where the model often copies or slightly modifies phrases from the prompt.
How It Works
When generating token t, the system:
- Looks back at the last n tokens generated (e.g., last 3-5 tokens)
- Searches the original prompt for matching n-grams
- Proposes the subsequent tokens from the prompt as draft candidates
- Verifies these candidates in parallel using the full model
Example Scenario
Prompt: "def calculate_sum(a, b):\n return a + b\n\ndef calculate_"
Model starts generating: "def calculate_"
The system finds "def calculate_" in the prompt, proposes "sum(a, b):" as draft tokens, then verifies them in one forward pass.
Why This Matters
No draft model needed: Zero additional memory or training overhead—just smart string matching. It's essentially "free" speculative decoding for the right workloads.
Works best when your application naturally involves:
- Code completion or refactoring
- Template-based generation
- Document revision tasks
- Multi-turn conversations referencing prior context
Key Takeaway: Prompt Lookup Decoding accelerates LLM inference by treating the input prompt as a draft model through n-gram matching, delivering speculative speedups without any additional models—ideal for code completion and high-overlap scenarios.