Course contentsShow
← AI Engineering
Lesson 138 of 1,8864. Prompt Engineering FoundationsFree lesson

Top-p (Nucleus) Sampling

Controlling diversity by sampling from the smallest set of tokens whose cumulative probability exceeds p.

Top-p (Nucleus) Sampling

What you'll learn: How to control output diversity by sampling from only the most probable tokens, ensuring coherent yet varied responses.

The Core Idea

While temperature controls overall randomness, top-p sampling (also called nucleus sampling) takes a different approach: it only considers the smallest group of tokens whose combined probabilities add up to p (a value between 0 and 1).

Think of it like a party invitation list. Instead of inviting everyone (high temperature) or just the VIP (low temperature), you invite people until you've filled p% of your venue capacity, starting with the most important guests first.

How It Works

When the model generates the next token, it:

  1. Ranks all possible tokens by probability
  2. Adds probabilities from the top, moving down the list
  3. Stops when the cumulative sum reaches p
  4. Randomly samples from only this nucleus of tokens

Example: With top_p=0.9, the model might consider the top 10 tokens that together make up 90% probability, ignoring hundreds of unlikely options. With top_p=0.5, it might only consider the top 3-5 most likely tokens.

Practical Impact

  • top_p=1.0: Consider all tokens (no filtering)
  • top_p=0.9: Balanced creativity and coherence (common default)
  • top_p=0.5: More focused, predictable outputs
  • top_p=0.1: Very conservative, similar to greedy decoding

Top-p works well with moderate temperature values. You're controlling which tokens are in the lottery, while temperature controls how the lottery is weighted.

Key Takeaway: Top-p sampling dynamically adjusts the candidate pool based on probability distribution, filtering out unlikely tokens while preserving natural variance in the most probable options.