Course contentsShow
← Machine Learning and Deep Learning
Lesson 2995 of 3,53865. LLM Inference EnginesFree lesson

Acceptance Rate and Expected Speedup

Mathematical analysis of how draft model quality affects acceptance rate, and calculating expected wall-clock speedup from speculative decoding.

Acceptance Rate and Expected Speedup

What you'll learn: How to mathematically analyze the relationship between draft model quality and the actual wall-clock speedup gained from speculative decoding.

The Quality-Speed Connection

Speculative decoding's effectiveness hinges on a critical metric: the acceptance rate — the fraction of draft tokens that pass verification and get accepted. A higher-quality draft model produces tokens that align more closely with what the target model would generate, leading to more acceptances per speculative batch.

The acceptance rate directly determines your speedup. If your draft model generates 5 tokens but only 2 are typically accepted, you're processing fewer tokens per iteration than if 4 out of 5 were accepted.

Calculating Expected Speedup

The expected speedup formula accounts for:

Expected tokens per iteration = 1 + (draft_length × acceptance_rate)

For example, with 4 draft tokens and 70% acceptance rate:

  • Expected accepted: 1 + (4 × 0.7) = 3.8 tokens per iteration
  • Versus baseline: 1 token per iteration
  • Theoretical speedup: 3.8×

However, wall-clock speedup is lower because:

  1. Draft generation takes time (though much less than target model)
  2. Verification requires one target model forward pass
  3. Memory bandwidth overhead

Realistic speedup ≈ (1 + draft_length × acceptance_rate) / (1 + draft_overhead_ratio)

If the draft model adds 0.1× overhead:

  • Speedup = 3.8 / 1.1 ≈ 3.45× actual speedup

The Trade-off

Larger, more accurate draft models increase acceptance rate but add overhead. Smaller draft models run faster but miss more predictions. The sweet spot maximizes this ratio.

Key Takeaway: Speculative decoding speedup depends on acceptance rate (determined by draft model quality) balanced against draft generation overhead — acceptance rates above 60-70% typically justify the technique's complexity.