Capability Degradation from RLHF
What you'll learn: How alignment training with RLHF can inadvertently reduce a language model's performance on tasks the reward model wasn't designed to evaluate.
The Core Problem
When we fine-tune large language models using Reinforcement Learning from Human Feedback (RLHF), we optimize them to maximize scores from a reward model. However, this reward model is trained on a specific distribution of human preferences—typically focused on helpfulness, harmlessness, and honesty for common user queries.
The catch: the model becomes very good at what the reward model measures, but may get worse at capabilities outside that distribution.
Why This Happens
Think of it like training a student solely for standardized tests. They'll excel at test-taking but might lose broader problem-solving skills or creativity not measured by the exam.
During RLHF, the model learns to avoid responses that score poorly with the reward model—even if those responses would be correct or useful for specialized tasks. The optimization pressure pushes the model toward "safe" outputs that reliably score well, potentially sacrificing:
- Specialized domain knowledge (technical, scientific, medical)
- Creative or unconventional reasoning
- Precise factual recall
- Performance on edge cases or unusual prompts
This is particularly problematic because the reward model was trained on limited human feedback data, which cannot cover the full breadth of human knowledge or use cases.
Connection to Previous Concepts
This relates directly to Goodhart's Law in RLHF and reward overoptimization: when you optimize too hard for a proxy metric (reward model score), you sacrifice performance on the true objective (general capability and usefulness).
Key Takeaway: RLHF can create a tension between alignment and capability—optimizing heavily for reward model approval may reduce the model's general-purpose competence on tasks outside the training distribution, requiring careful balancing during deployment.