Repetition Penalty and Diversity
You'll learn how to prevent language models from repeating themselves by penalizing previously generated tokens.
The Problem with Pure Sampling
Even with temperature and top-k/top-p sampling, autoregressive models sometimes get stuck in repetitive loops—generating the same words or phrases over and over. This happens because once a token appears, similar contexts might keep favoring it during next-token prediction.
How Repetition Penalty Works
Repetition penalty downweights the probability of tokens that have already been generated in the current sequence. Before sampling your next token, you divide the logits (pre-softmax scores) of previously seen tokens by a penalty factor (typically 1.0–1.5).
Analogy: Imagine telling a story where every time you use a word, that word becomes slightly less appealing to use again—like putting already-played cards at the bottom of your deck. This forces variety.
The Mechanism
- Generate several tokens autoregressively
- Before sampling the next token, identify all tokens already in your output
- Reduce their logits by dividing by the penalty value (e.g., 1.2)
- Apply softmax and sample as usual (with temperature, top-k, etc.)
Higher penalties = more diversity, but too much can make output incoherent or unnatural.
Balancing Act
- No penalty (1.0): Natural but possibly repetitive
- Moderate penalty (1.1–1.3): Reduces loops while staying coherent
- High penalty (>1.5): Very diverse but may sound forced or random
Repetition penalty works alongside your other sampling strategies—temperature controls randomness, top-p controls variety in the candidate pool, and repetition penalty specifically targets redundancy.
Key Takeaway: Repetition penalty reduces output redundancy by making previously generated tokens less likely to be selected again, improving diversity without requiring model retraining.