Temperature Sampling
What you'll learn: How to control the randomness and creativity of text generation by adjusting the "temperature" parameter that scales probabilities before sampling.
The Core Idea
When generating text autoregressively (one token at a time), we don't always want the most probable token — that can lead to repetitive, boring output. Temperature sampling gives us a knob to dial between predictable and creative generation.
Here's how it works: Before applying softmax to convert logits (raw model outputs) into probabilities, we divide the logits by a temperature value T. This scaling changes the shape of the probability distribution.
How Temperature Changes Behavior
Think of temperature like adjusting the confidence of a shy vs. bold speaker:
- T = 1.0 (baseline): Use the model's original probability distribution — no change
- T < 1.0 (e.g., 0.5): Makes high-probability tokens even more likely; the model becomes more "confident" and conservative. Output is more focused and deterministic.
- T > 1.0 (e.g., 1.5): Flattens the distribution, giving lower-probability tokens more chance. The model becomes more "creative" and random, exploring diverse possibilities.
- T → 0: Approaches greedy decoding (always pick the most likely token)
- T → ∞: All tokens become equally likely (pure randomness)
The Math Briefly
Original probability: P(token) = softmax(logits)
With temperature: P(token) = softmax(logits / T)
Lower temperature sharpens peaks; higher temperature smooths them out.
Key Takeaway: Temperature sampling lets you control the randomness-creativity trade-off in text generation by scaling logits before computing probabilities — lower temperatures produce safer, more predictable text while higher temperatures encourage diversity and exploration.