Course contentsShow
← Machine Learning and Deep Learning
Lesson 1313 of 3,53829. Core NLP TasksFree lesson

Sampling-Based Decoding Methods

Explore temperature scaling, top-k sampling, and top-p (nucleus) sampling to introduce controlled randomness in generation.

Sampling-Based Decoding Methods

What you'll learn: How to introduce controlled randomness into text generation using temperature scaling, top-k sampling, and top-p (nucleus) sampling.

The Problem with Pure Randomness

After covering greedy and beam search, you might wonder: what if we want more creativity in generated text? We could just sample randomly from the entire vocabulary at each step, but that often produces nonsense. Sampling-based methods give us the best of both worlds: randomness for variety, but constrained to keep outputs sensible.

Temperature Scaling

Temperature reshapes the probability distribution before sampling. Think of it like adjusting the "confidence" of your model:

  • Temperature = 1.0: Use raw probabilities unchanged
  • Temperature < 1.0 (e.g., 0.5): Make the model more confident—sharp peaks, favoring top choices (less random)
  • Temperature > 1.0 (e.g., 1.5): Flatten the distribution—more uniform, allowing unlikely tokens (more random)

Analogy: Like adjusting the heat on a stovetop—lower temperature makes outcomes more predictable (water simmers gently), higher temperature introduces more chaos (rolling boil).

Top-k Sampling

Only consider the k most likely tokens at each step, then sample from those. If k=50, you ignore all but the top 50 options, redistributing their probabilities.

Benefit: Eliminates long-tail nonsense tokens while maintaining variety.

Top-p (Nucleus) Sampling

Instead of a fixed number, select the smallest set of tokens whose cumulative probability exceeds p (e.g., p=0.9). This adapts dynamically—sometimes choosing 10 tokens, sometimes 100, depending on how peaked the distribution is.

Benefit: More flexible than top-k; captures high-confidence situations and uncertain ones naturally.

Key Takeaway: Temperature, top-k, and top-p sampling control randomness in generation by selectively filtering or reshaping token probabilities, enabling creative yet coherent outputs beyond deterministic methods.