Multi-Token Prediction for Speculation
What you'll learn: How to train a single model to predict multiple future tokens at once, eliminating the need for separate draft models in speculative decoding.
The Core Innovation
Traditional speculative decoding requires two models: a fast draft model and a slower target model. But what if your main model could generate its own "drafts" internally? Multi-token prediction trains a language model with additional prediction heads that simultaneously forecast not just the next token, but the next 2, 3, or even more tokens ahead.
How It Works
Instead of one output layer predicting token[t+1], you add parallel heads predicting token[t+2], token[t+3], etc. During training, the model learns patterns like "after 'New' comes 'York', and after that probably 'City'." At inference time, these predictions become your draft tokens—generated in a single forward pass by the same model that will verify them.
The key advantage: No separate draft model means no architecture mismatch, no distillation overhead, and no coordination complexity. The model essentially speculates against itself.
Self-Speculation in Action
When generating text, the model produces multiple candidate continuations simultaneously. It then verifies these candidates using its own autoregressive logic (similar to the verification step you learned in speculative decoding). Accepted tokens move forward; rejected ones fall back to single-token generation.
This approach bridges self-speculative techniques with the multi-head prediction paradigm, offering 1.5-3x speedups depending on how predictable the text is.
Key Takeaway: Multi-token prediction transforms a single model into both draft generator and verifier by training it to forecast several tokens ahead simultaneously, achieving speculation without architectural duplication.