Course contentsShow
← Machine Learning and Deep Learning
Lesson 1772 of 3,53838. Instruction Tuning and AlignmentFree lesson

KL Divergence Penalty: Why It Matters

How KL regularization prevents reward hacking, mode collapse, and maintains language model capabilities.

KL Divergence Penalty: Why It Matters

You'll learn why the KL divergence penalty in RLHF prevents your language model from going off the rails while chasing high rewards.

The Core Problem

When fine-tuning your policy model with reinforcement learning (from lesson 1771's RLHF objective), you're optimizing the model to maximize rewards from your reward model. But here's the danger: without constraints, your model might discover bizarre "reward hacks" — nonsensical outputs that accidentally score high rewards but destroy the model's actual language capabilities.

Think of it like training a dog with treats. If you only reward the dog without any boundaries, it might start doing extreme behaviors that technically earn treats but aren't what you wanted (like knocking over the treat jar).

How KL Divergence Protects You

The KL divergence penalty measures how much your policy model's outputs diverge from the reference model (your original SFT model). By adding a penalty term proportional to this divergence, you're essentially saying: "Yes, maximize rewards, but stay close to your original behavior."

This penalty prevents three critical failures:

  1. Reward hacking: Exploiting unintended patterns the reward model learned
  2. Mode collapse: Generating repetitive or formulaic responses that score well
  3. Capability degradation: Losing coherence, factuality, or fluency

The KL penalty acts like a leash, letting your model explore better behaviors while staying grounded in its pretrained knowledge.

Key Takeaway: The KL divergence penalty balances reward optimization with language model stability, ensuring your RLHF-trained model improves helpfulness without sacrificing coherence or exploiting reward model weaknesses.