Course contentsShow
← Machine Learning and Deep Learning
Lesson 1195 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Repetition Penalty and Diversity

Penalizing previously generated tokens to reduce repetition and increase output diversity.

Repetition Penalty and Diversity

You'll learn how to prevent language models from repeating themselves by penalizing previously generated tokens.

The Problem with Pure Sampling

Even with temperature and top-k/top-p sampling, autoregressive models sometimes get stuck in repetitive loops—generating the same words or phrases over and over. This happens because once a token appears, similar contexts might keep favoring it during next-token prediction.

How Repetition Penalty Works

Repetition penalty downweights the probability of tokens that have already been generated in the current sequence. Before sampling your next token, you divide the logits (pre-softmax scores) of previously seen tokens by a penalty factor (typically 1.0–1.5).

Analogy: Imagine telling a story where every time you use a word, that word becomes slightly less appealing to use again—like putting already-played cards at the bottom of your deck. This forces variety.

The Mechanism

  1. Generate several tokens autoregressively
  2. Before sampling the next token, identify all tokens already in your output
  3. Reduce their logits by dividing by the penalty value (e.g., 1.2)
  4. Apply softmax and sample as usual (with temperature, top-k, etc.)

Higher penalties = more diversity, but too much can make output incoherent or unnatural.

Balancing Act

  • No penalty (1.0): Natural but possibly repetitive
  • Moderate penalty (1.1–1.3): Reduces loops while staying coherent
  • High penalty (>1.5): Very diverse but may sound forced or random

Repetition penalty works alongside your other sampling strategies—temperature controls randomness, top-p controls variety in the candidate pool, and repetition penalty specifically targets redundancy.

Key Takeaway: Repetition penalty reduces output redundancy by making previously generated tokens less likely to be selected again, improving diversity without requiring model retraining.