Acceptance Rate and Expected Speedup
What you'll learn: How to mathematically analyze the relationship between draft model quality and the actual wall-clock speedup gained from speculative decoding.
The Quality-Speed Connection
Speculative decoding's effectiveness hinges on a critical metric: the acceptance rate — the fraction of draft tokens that pass verification and get accepted. A higher-quality draft model produces tokens that align more closely with what the target model would generate, leading to more acceptances per speculative batch.
The acceptance rate directly determines your speedup. If your draft model generates 5 tokens but only 2 are typically accepted, you're processing fewer tokens per iteration than if 4 out of 5 were accepted.
Calculating Expected Speedup
The expected speedup formula accounts for:
Expected tokens per iteration = 1 + (draft_length × acceptance_rate)
For example, with 4 draft tokens and 70% acceptance rate:
- Expected accepted: 1 + (4 × 0.7) = 3.8 tokens per iteration
- Versus baseline: 1 token per iteration
- Theoretical speedup: 3.8×
However, wall-clock speedup is lower because:
- Draft generation takes time (though much less than target model)
- Verification requires one target model forward pass
- Memory bandwidth overhead
Realistic speedup ≈ (1 + draft_length × acceptance_rate) / (1 + draft_overhead_ratio)
If the draft model adds 0.1× overhead:
- Speedup = 3.8 / 1.1 ≈ 3.45× actual speedup
The Trade-off
Larger, more accurate draft models increase acceptance rate but add overhead. Smaller draft models run faster but miss more predictions. The sweet spot maximizes this ratio.
Key Takeaway: Speculative decoding speedup depends on acceptance rate (determined by draft model quality) balanced against draft generation overhead — acceptance rates above 60-70% typically justify the technique's complexity.