Course contentsShow
← AI Engineering
Lesson 139 of 1,8864. Prompt Engineering FoundationsFree lesson

Top-k Sampling

Limiting token selection to the k most probable next tokens to balance coherence and variety.

Top-k Sampling

What you'll learn: How to control text generation by limiting the model's next-token choices to only the k most likely options.

The Core Idea

When an LLM generates text, it produces a probability distribution over thousands of possible next tokens. Top-k sampling restricts this choice by keeping only the k highest-probability tokens and redistributing their probabilities before sampling.

Think of it like a restaurant menu: instead of overwhelming yourself with 100 dishes (many of which sound terrible), you ask the waiter for their "top 5 recommendations" and choose randomly from those. You get variety without chaos.

How It Works

  1. The model calculates probabilities for all possible next tokens
  2. Sort tokens by probability (highest to lowest)
  3. Keep only the top k tokens
  4. Discard everything else (set their probability to zero)
  5. Renormalize the remaining k probabilities so they sum to 1
  6. Sample randomly from these k candidates

Typical Values

  • k=1: Greedy decoding (always picks the most likely token—no randomness)
  • k=10-50: Balanced coherence and creativity
  • k=100+: More variety, but can introduce less relevant tokens

Top-k vs Top-p

You learned Top-p (nucleus) sampling in the previous lesson. The key difference:

  • Top-p: Dynamic—keeps however many tokens needed to reach a probability threshold
  • Top-k: Fixed—always keeps exactly k tokens, regardless of their probability distribution

Top-k is simpler but less adaptive than top-p. Many APIs let you use both together for fine-grained control.

Key Takeaway: Top-k sampling balances coherence and variety by restricting next-token selection to a fixed number of top candidates, preventing the model from choosing improbable words while maintaining creative freedom.