Draft Model Requirements and Selection
What you'll learn: How to choose the right draft model for speculative decoding by balancing size, compatibility, and the speed-accuracy tradeoff.
The Draft Model's Critical Role
In speculative decoding, the draft model generates candidate tokens quickly, which the larger target model then verifies. Think of the draft model as a quick sketching artist proposing ideas, while the target model is the master painter who refines or rejects them. Choosing the wrong draft model is like hiring a sketch artist who's either too slow or whose style completely mismatches the painter's — you lose the speedup benefit.
Three Key Selection Criteria
Size Constraints
The draft model must be significantly smaller than the target model — typically 10-100x fewer parameters. A 70B parameter target might pair with a 7B or even 1B draft model. If the draft is too large, its generation speed won't offset verification overhead, eliminating the performance gain.
Architectural Compatibility
The draft and target models should share similar tokenization, vocabulary, and output distributions. Mismatched architectures lead to poor speculation accuracy — the draft proposes tokens the target would never generate. Ideally, use models from the same family (e.g., Llama 7B drafting for Llama 70B) or models trained on similar data distributions.
Speed-Accuracy Tradeoff
This is the balancing act: faster draft models propose more tokens per second, but less accurate ones get rejected more often. A tiny but inaccurate draft might speculate 10 tokens but only 2 get accepted. A moderately sized, well-aligned draft might propose 5 tokens with 4 accepted — yielding better overall throughput. You want high acceptance rate while maintaining speed advantage.
Key Takeaway: Select draft models that are much smaller than your target, architecturally compatible for high acceptance rates, and optimized for the sweet spot where speculation speed gains exceed verification costs.