Course contentsShow
← Machine Learning and Deep Learning
Lesson 1792 of 3,53838. Instruction Tuning and AlignmentFree lesson

KL Divergence Penalty in LLM Training

Using KL divergence from the reference model to keep the policy close to pretrained capabilities.

KL Divergence Penalty in LLM Training

What You'll Learn

You'll understand how the KL divergence penalty keeps your fine-tuned model tethered to its pretrained knowledge during RLHF.

The Core Problem

During RLHF, your policy model learns to maximize rewards from the reward model. Without constraints, it might discover "shortcuts"—generating outputs that score high rewards but are nonsensical, repetitive, or completely unlike natural language. This is reward hacking in action.

How KL Divergence Acts as an Anchor

The KL divergence penalty measures how different your current policy's token predictions are from the original reference model (typically the SFT model before RL training). Think of it like a leash: it allows the model to explore better behaviors to earn rewards, but pulls it back if it strays too far from its pretrained capabilities.

Mathematically, for each generated token, we compare:

  • The probability distribution from your policy model (being optimized)
  • The probability distribution from your reference model (frozen)

The KL divergence quantifies this difference. A high KL value means the policy is predicting very differently from the reference—potentially "forgetting" its language modeling skills.

The Balancing Act

The RLHF objective combines two terms:

  1. Reward maximization: Make the reward model happy
  2. KL penalty: Stay close to the reference model

The penalty term is typically: β * KL(policy || reference), where β controls how tightly you constrain the policy. Higher β = tighter leash, more conservative updates. Lower β = more freedom to optimize for rewards, but higher risk of drift.

This ensures your model improves alignment while preserving fluency, coherence, and general knowledge.

Key Takeaway: The KL divergence penalty prevents your policy from drifting too far from pretrained capabilities during reward optimization, balancing alignment improvements with language quality preservation.