Course contentsShow
← Machine Learning and Deep Learning
Lesson 2999 of 3,53865. LLM Inference EnginesFree lesson

Prompt Lookup Decoding

Leveraging n-gram matches in the prompt as draft candidates for scenarios with high input-output overlap like code completion.

Prompt Lookup Decoding

What you'll learn: How to exploit n-gram overlap between input prompts and expected outputs to accelerate LLM inference without requiring a separate draft model.

The Core Idea

Prompt Lookup Decoding is a clever variant of speculative decoding that uses the prompt itself as a source of draft tokens. Instead of maintaining a separate draft model, it searches for n-gram matches between recently generated tokens and segments of the original input prompt.

This technique shines in scenarios with high input-output overlap—think code completion, document editing, or chat contexts where the model often copies or slightly modifies phrases from the prompt.

How It Works

When generating token t, the system:

  1. Looks back at the last n tokens generated (e.g., last 3-5 tokens)
  2. Searches the original prompt for matching n-grams
  3. Proposes the subsequent tokens from the prompt as draft candidates
  4. Verifies these candidates in parallel using the full model

Example Scenario

Prompt: "def calculate_sum(a, b):\n    return a + b\n\ndef calculate_"
Model starts generating: "def calculate_"

The system finds "def calculate_" in the prompt, proposes "sum(a, b):" as draft tokens, then verifies them in one forward pass.

Why This Matters

No draft model needed: Zero additional memory or training overhead—just smart string matching. It's essentially "free" speculative decoding for the right workloads.

Works best when your application naturally involves:

  • Code completion or refactoring
  • Template-based generation
  • Document revision tasks
  • Multi-turn conversations referencing prior context

Key Takeaway: Prompt Lookup Decoding accelerates LLM inference by treating the input prompt as a draft model through n-gram matching, delivering speculative speedups without any additional models—ideal for code completion and high-overlap scenarios.