Top-p (Nucleus) Sampling
What you'll learn: How to control output diversity by sampling from only the most probable tokens, ensuring coherent yet varied responses.
The Core Idea
While temperature controls overall randomness, top-p sampling (also called nucleus sampling) takes a different approach: it only considers the smallest group of tokens whose combined probabilities add up to p (a value between 0 and 1).
Think of it like a party invitation list. Instead of inviting everyone (high temperature) or just the VIP (low temperature), you invite people until you've filled p% of your venue capacity, starting with the most important guests first.
How It Works
When the model generates the next token, it:
- Ranks all possible tokens by probability
- Adds probabilities from the top, moving down the list
- Stops when the cumulative sum reaches
p - Randomly samples from only this nucleus of tokens
Example: With top_p=0.9, the model might consider the top 10 tokens that together make up 90% probability, ignoring hundreds of unlikely options. With top_p=0.5, it might only consider the top 3-5 most likely tokens.
Practical Impact
top_p=1.0: Consider all tokens (no filtering)top_p=0.9: Balanced creativity and coherence (common default)top_p=0.5: More focused, predictable outputstop_p=0.1: Very conservative, similar to greedy decoding
Top-p works well with moderate temperature values. You're controlling which tokens are in the lottery, while temperature controls how the lottery is weighted.
Key Takeaway: Top-p sampling dynamically adjusts the candidate pool based on probability distribution, filtering out unlikely tokens while preserving natural variance in the most probable options.