Course contentsShow
← Machine Learning and Deep Learning
Lesson 2996 of 3,53865. LLM Inference EnginesFree lesson

Temperature and Sampling in Speculative Decoding

Handling non-greedy decoding: modified rejection sampling to maintain exact output distribution match with temperature and top-p sampling.

Temperature and Sampling in Speculative Decoding

What You'll Learn

You'll learn how speculative decoding maintains exact output distribution matching when using non-greedy sampling methods like temperature scaling and top-p sampling.

The Challenge with Non-Greedy Decoding

In greedy decoding, speculative decoding is straightforward: verify each draft token matches what the target model would have chosen deterministically. But what happens when you use temperature or top-p sampling, where outputs are probabilistic?

The naive approach—simply accepting draft tokens if they match—would change the output distribution compared to running the target model alone. This breaks a core promise: speculative decoding must produce identical results to standard autoregressive decoding, just faster.

Modified Rejection Sampling

The solution uses a clever modified rejection sampling algorithm:

  1. The draft model samples a token using temperature/top-p
  2. The target model computes probabilities for all candidates in parallel
  3. For each draft token, calculate an acceptance probability: min(1, p_target(token) / p_draft(token))
  4. Accept the token with this probability; if rejected, sample a new token from an adjusted distribution: max(0, p_target - p_draft) (renormalized)

This ensures the final distribution exactly matches what you'd get from the target model alone, maintaining properties like diversity and creativity controlled by temperature.

Why It Works

Think of it like a quality control process: you tentatively accept draft suggestions, but correct for any bias the draft model introduced. The math guarantees that after rejection sampling, it's as if the target model sampled directly—no statistical difference in outputs.

Key Takeaway: Modified rejection sampling in speculative decoding preserves exact output distributions for temperature and top-p sampling, ensuring speedups don't compromise generation quality or diversity.