Speculative Decoding: Core Intuition
What you'll learn: How to accelerate LLM inference by generating multiple candidate tokens in parallel, then verifying them in one pass.
The Core Idea
Remember how autoregressive decoding generates tokens one at a time, each requiring a full forward pass through the large model? Speculative decoding breaks this sequential bottleneck with a clever trick: use a small, fast draft model to generate several candidate tokens speculatively, then use the larger target model to verify all candidates in a single forward pass.
Think of it like a student (draft model) solving multiple practice problems quickly, then having a teacher (target model) grade them all at once. Instead of waiting for the teacher to solve each problem step-by-step, we leverage the fast student's guesses and batch the verification.
How It Works
- Draft Phase: A smaller, faster model generates k candidate tokens sequentially (e.g., 4-8 tokens ahead)
- Verification Phase: The large target model processes all candidates in one parallel forward pass
- Acceptance: The target model accepts correct predictions and rejects the first wrong one, then continues from there
The key insight: even if the draft model is less accurate, as long as it's right sometimes, you save computation because verifying multiple tokens together is cheaper than generating them one-by-one with the large model.
Memory and Efficiency
The draft model can share the same KV cache management techniques (like paged attention) you've already learned. The target model verification pass processes multiple positions simultaneously, amortizing the cost across several tokens.
Key Takeaway: Speculative decoding trades draft model computation for target model computation, achieving speedups when the draft model is much faster and reasonably accurate, by verifying multiple candidate tokens in parallel rather than generating them autoregressively.