Top-k and Top-p (Nucleus) Sampling
You'll learn how to improve text generation quality by intelligently restricting which tokens the model can choose from during sampling.
The Problem with Pure Temperature Sampling
Temperature sampling (lesson 1193) adjusts probabilities across all vocabulary tokens. But what if the model assigns tiny probabilities to hundreds of nonsensical words? Even with temperature adjustment, we might still occasionally sample gibberish.
Top-k Sampling: The Fixed Cut-off
Top-k sampling restricts selection to only the k most probable tokens at each step.
Example: If k=50, the model considers only its top 50 predicted words, redistributes their probabilities (renormalizing), then samples from this reduced set. The remaining vocabulary is ignored.
Analogy: Like asking a chef for their top 10 dish recommendations instead of choosing randomly from the entire menu—you avoid the weird experimental stuff.
Limitation: A fixed k doesn't adapt. Sometimes the model is very confident (one clear best token), other times uncertain (many reasonable options). Fixed k=50 might be too restrictive when confident, too permissive when uncertain.
Top-p (Nucleus) Sampling: The Adaptive Cut-off
Top-p sampling (or nucleus sampling) solves this by using a probability threshold instead of a fixed number. You set p (e.g., 0.9), and the model selects the smallest set of tokens whose cumulative probability exceeds p.
Example: If the top 3 tokens have probabilities [0.5, 0.3, 0.15], their sum is 0.95. With p=0.9, we'd sample from just these 3. If probabilities were more spread out, we might need 20 tokens to reach 0.9.
Why it's better: The "nucleus" size adapts to the model's confidence, maintaining both quality and diversity.
Key Takeaway: Top-k uses a fixed number of candidates; top-p (nucleus sampling) dynamically adjusts the candidate pool based on cumulative probability, producing more natural and context-appropriate text generation.