Sampling-Based Decoding Methods
What you'll learn: How to introduce controlled randomness into text generation using temperature scaling, top-k sampling, and top-p (nucleus) sampling.
The Problem with Pure Randomness
After covering greedy and beam search, you might wonder: what if we want more creativity in generated text? We could just sample randomly from the entire vocabulary at each step, but that often produces nonsense. Sampling-based methods give us the best of both worlds: randomness for variety, but constrained to keep outputs sensible.
Temperature Scaling
Temperature reshapes the probability distribution before sampling. Think of it like adjusting the "confidence" of your model:
- Temperature = 1.0: Use raw probabilities unchanged
- Temperature < 1.0 (e.g., 0.5): Make the model more confident—sharp peaks, favoring top choices (less random)
- Temperature > 1.0 (e.g., 1.5): Flatten the distribution—more uniform, allowing unlikely tokens (more random)
Analogy: Like adjusting the heat on a stovetop—lower temperature makes outcomes more predictable (water simmers gently), higher temperature introduces more chaos (rolling boil).
Top-k Sampling
Only consider the k most likely tokens at each step, then sample from those. If k=50, you ignore all but the top 50 options, redistributing their probabilities.
Benefit: Eliminates long-tail nonsense tokens while maintaining variety.
Top-p (Nucleus) Sampling
Instead of a fixed number, select the smallest set of tokens whose cumulative probability exceeds p (e.g., p=0.9). This adapts dynamically—sometimes choosing 10 tokens, sometimes 100, depending on how peaked the distribution is.
Benefit: More flexible than top-k; captures high-confidence situations and uncertain ones naturally.
Key Takeaway: Temperature, top-k, and top-p sampling control randomness in generation by selectively filtering or reshaping token probabilities, enabling creative yet coherent outputs beyond deterministic methods.