Course contentsShow
← Machine Learning and Deep Learning
Lesson 3441 of 3,53875. LLM Safety and Alignment ChallengesFree lesson

Mode Collapse and Response Diversity

How RLHF can lead to repetitive or formulaic outputs as models converge to high-reward but stereotyped responses.

Mode Collapse and Response Diversity

What you'll learn: How RLHF can cause models to produce repetitive, formulaic outputs by converging on high-reward but stereotyped responses.

The Convergence Problem

When you optimize a language model using RLHF, the reward model assigns scores to different responses. Over time, the policy model learns which patterns consistently earn high rewards. But here's the catch: the model may discover a few "safe" response templates that always score well, then overuse them.

This phenomenon is called mode collapse—the model's output distribution collapses from diverse, creative responses down to a narrow set of high-reward patterns.

Why It Happens

Think of it like a student who discovers that starting every essay with "In today's society..." and using five-paragraph format always gets a B+. They stop experimenting with other styles, even though those might sometimes earn an A. The reward signal, while positive, inadvertently punishes diversity.

In RLHF, if certain phrasings ("As an AI assistant, I'd be happy to help...") or structures (numbered lists, cautious hedging) consistently score well with the reward model, the policy learns to default to these patterns. The model becomes risk-averse, avoiding novel responses that might score lower, even if they'd be more appropriate for specific contexts.

The Cost to Response Diversity

The result? Your LLM might:

  • Start every answer with the same greeting
  • Overuse certain sentence structures
  • Provide generically "helpful" but bland responses
  • Lose the creative variation present before RLHF fine-tuning

This reduces the model's ability to match its tone and style to different users, contexts, or creative tasks.

Key Takeaway: Mode collapse in RLHF causes models to converge on stereotyped, high-reward response patterns at the expense of diversity, making outputs feel repetitive and formulaic even when technically correct.