KL Divergence Penalty: Why It Matters
You'll learn why the KL divergence penalty in RLHF prevents your language model from going off the rails while chasing high rewards.
The Core Problem
When fine-tuning your policy model with reinforcement learning (from lesson 1771's RLHF objective), you're optimizing the model to maximize rewards from your reward model. But here's the danger: without constraints, your model might discover bizarre "reward hacks" — nonsensical outputs that accidentally score high rewards but destroy the model's actual language capabilities.
Think of it like training a dog with treats. If you only reward the dog without any boundaries, it might start doing extreme behaviors that technically earn treats but aren't what you wanted (like knocking over the treat jar).
How KL Divergence Protects You
The KL divergence penalty measures how much your policy model's outputs diverge from the reference model (your original SFT model). By adding a penalty term proportional to this divergence, you're essentially saying: "Yes, maximize rewards, but stay close to your original behavior."
This penalty prevents three critical failures:
- Reward hacking: Exploiting unintended patterns the reward model learned
- Mode collapse: Generating repetitive or formulaic responses that score well
- Capability degradation: Losing coherence, factuality, or fluency
The KL penalty acts like a leash, letting your model explore better behaviors while staying grounded in its pretrained knowledge.
Key Takeaway: The KL divergence penalty balances reward optimization with language model stability, ensuring your RLHF-trained model improves helpfulness without sacrificing coherence or exploiting reward model weaknesses.