Course contentsShow
← Machine Learning and Deep Learning
Lesson 1323 of 3,53829. Core NLP TasksFree lesson

Repetition and Degeneration Problems

Understand why models produce repetitive text and learn mitigation strategies like repetition penalties and diverse beam search.

Repetition and Degeneration Problems

What you'll learn: Why language models sometimes produce repetitive or boring text, and how to fix it.

The Problem: When Models Get Stuck

When generating text, neural language models sometimes fall into predictable patterns—repeating the same phrases, words, or sentence structures over and over. This is called text degeneration or repetition.

Think of it like a conversation partner who keeps saying "you know what I mean?" after every sentence. Technically correct, but frustrating and unnatural.

Why Does This Happen?

Language models predict the most probable next token based on context. Sometimes, this probability distribution becomes too "peaky"—one option dominates overwhelmingly. Once a model starts repeating, that repetition becomes part of the context, making further repetition even more likely. It's a feedback loop.

For example:

"The cat sat on the mat. The cat sat on the mat. The cat sat on the mat..."

This often happens with greedy decoding or narrow beam search, where you're always picking high-probability tokens without enough diversity.

Mitigation Strategies

Repetition Penalty: Artificially reduce the probability of tokens that have already appeared in the generated sequence. You apply a multiplicative penalty (typically 1.2–2.0) to discourage reuse.

Diverse Beam Search: Instead of maintaining multiple beams that converge on similar outputs, enforce diversity by dividing beams into groups and penalizing similarity within groups. This encourages exploring different generation paths.

No-Repeat N-grams: Block the model from generating n-grams (like bigrams or trigrams) that have already appeared.

These techniques trade off some likelihood score for more natural, varied output.

Key Takeaway: Repetition occurs when models over-rely on high-probability tokens; mitigation strategies like repetition penalties and diverse beam search break the feedback loop by encouraging exploration and penalizing redundancy.