Course contentsShow
← Machine Learning and Deep Learning
Lesson 3000 of 3,53865. LLM Inference EnginesFree lesson

Multi-Token Prediction for Speculation

Training models to predict multiple future tokens per forward pass, enabling self-speculation without separate draft models.

Multi-Token Prediction for Speculation

What you'll learn: How to train a single model to predict multiple future tokens at once, eliminating the need for separate draft models in speculative decoding.

The Core Innovation

Traditional speculative decoding requires two models: a fast draft model and a slower target model. But what if your main model could generate its own "drafts" internally? Multi-token prediction trains a language model with additional prediction heads that simultaneously forecast not just the next token, but the next 2, 3, or even more tokens ahead.

How It Works

Instead of one output layer predicting token[t+1], you add parallel heads predicting token[t+2], token[t+3], etc. During training, the model learns patterns like "after 'New' comes 'York', and after that probably 'City'." At inference time, these predictions become your draft tokens—generated in a single forward pass by the same model that will verify them.

The key advantage: No separate draft model means no architecture mismatch, no distillation overhead, and no coordination complexity. The model essentially speculates against itself.

Self-Speculation in Action

When generating text, the model produces multiple candidate continuations simultaneously. It then verifies these candidates using its own autoregressive logic (similar to the verification step you learned in speculative decoding). Accepted tokens move forward; rejected ones fall back to single-token generation.

This approach bridges self-speculative techniques with the multi-head prediction paradigm, offering 1.5-3x speedups depending on how predictable the text is.

Key Takeaway: Multi-token prediction transforms a single model into both draft generator and verifier by training it to forecast several tokens ahead simultaneously, achieving speculation without architectural duplication.