KL Divergence Penalty in LLM Training
What You'll Learn
You'll understand how the KL divergence penalty keeps your fine-tuned model tethered to its pretrained knowledge during RLHF.
The Core Problem
During RLHF, your policy model learns to maximize rewards from the reward model. Without constraints, it might discover "shortcuts"—generating outputs that score high rewards but are nonsensical, repetitive, or completely unlike natural language. This is reward hacking in action.
How KL Divergence Acts as an Anchor
The KL divergence penalty measures how different your current policy's token predictions are from the original reference model (typically the SFT model before RL training). Think of it like a leash: it allows the model to explore better behaviors to earn rewards, but pulls it back if it strays too far from its pretrained capabilities.
Mathematically, for each generated token, we compare:
- The probability distribution from your policy model (being optimized)
- The probability distribution from your reference model (frozen)
The KL divergence quantifies this difference. A high KL value means the policy is predicting very differently from the reference—potentially "forgetting" its language modeling skills.
The Balancing Act
The RLHF objective combines two terms:
- Reward maximization: Make the reward model happy
- KL penalty: Stay close to the reference model
The penalty term is typically: β * KL(policy || reference), where β controls how tightly you constrain the policy. Higher β = tighter leash, more conservative updates. Lower β = more freedom to optimize for rewards, but higher risk of drift.
This ensures your model improves alignment while preserving fluency, coherence, and general knowledge.
Key Takeaway: The KL divergence penalty prevents your policy from drifting too far from pretrained capabilities during reward optimization, balancing alignment improvements with language quality preservation.