Top-k Sampling
What you'll learn: How to control text generation by limiting the model's next-token choices to only the k most likely options.
The Core Idea
When an LLM generates text, it produces a probability distribution over thousands of possible next tokens. Top-k sampling restricts this choice by keeping only the k highest-probability tokens and redistributing their probabilities before sampling.
Think of it like a restaurant menu: instead of overwhelming yourself with 100 dishes (many of which sound terrible), you ask the waiter for their "top 5 recommendations" and choose randomly from those. You get variety without chaos.
How It Works
- The model calculates probabilities for all possible next tokens
- Sort tokens by probability (highest to lowest)
- Keep only the top k tokens
- Discard everything else (set their probability to zero)
- Renormalize the remaining k probabilities so they sum to 1
- Sample randomly from these k candidates
Typical Values
- k=1: Greedy decoding (always picks the most likely token—no randomness)
- k=10-50: Balanced coherence and creativity
- k=100+: More variety, but can introduce less relevant tokens
Top-k vs Top-p
You learned Top-p (nucleus) sampling in the previous lesson. The key difference:
- Top-p: Dynamic—keeps however many tokens needed to reach a probability threshold
- Top-k: Fixed—always keeps exactly k tokens, regardless of their probability distribution
Top-k is simpler but less adaptive than top-p. Many APIs let you use both together for fine-grained control.
Key Takeaway: Top-k sampling balances coherence and variety by restricting next-token selection to a fixed number of top candidates, preventing the model from choosing improbable words while maintaining creative freedom.