Greedy Decoding
You'll learn how the simplest autoregressive sampling strategy picks the most likely token at every generation step.
What Is Greedy Decoding?
When your language model generates text autoregressively (one token at a time), greedy decoding is the straightforward strategy of always selecting the single highest-probability token from the model's output distribution at each step.
Think of it like choosing from a restaurant menu by always picking the dish with the highest rating, never considering alternatives. At time step t, you look at all possible next tokens, find the one with maximum probability, lock it in, feed it back as input, and repeat for step t+1.
How It Works
- Your model outputs a probability distribution over the vocabulary
- Apply
argmax()to find the token with highest probability - Append that token to your sequence
- Feed the updated sequence back in
- Repeat until you hit a stop condition (e.g.,
[END]token or max length)
Key Characteristics
Speed: Greedy decoding is extremely fast—just one forward pass per token, no search overhead.
Determinism: Given the same starting prompt and model, you'll always get the exact same output. No randomness involved.
Limitations: It can miss better overall sequences because it never reconsiders earlier choices. A locally optimal token might lead to a poor continuation later, but greedy decoding has no mechanism to backtrack or explore alternatives.
Key Takeaway: Greedy decoding selects the highest-probability token at each step—fast and deterministic, but unable to explore alternative paths that might produce better complete sequences.