The RLHF Objective Function
What you'll learn: How RLHF balances reward maximization with staying close to a reference model to create stable, aligned language models.
The Core Challenge
After training a reward model, we want to use reinforcement learning to optimize our policy model to maximize predicted rewards. But here's the problem: if we only maximize reward, the policy might exploit quirks in the reward model, generating outputs that score high but are nonsensical or harmful. It's like a student who figures out how to game a test without actually learning the material.
The Solution: A Balanced Objective
The RLHF objective function combines two competing goals:
- Maximize expected reward from the reward model
- Stay close to the reference model (usually the SFT model)
Mathematically, the objective looks like:
objective = E[reward(x, y)] - β * KL(π_θ || π_ref)
Where:
reward(x, y)is the reward model's score for promptxand responseyKL(π_θ || π_ref)is the KL divergence measuring how much the policyπ_θdiffers from the referenceπ_refβis a hyperparameter controlling the trade-off
Why KL Divergence Matters
Think of the reference model as a safety anchor. The KL divergence penalty prevents the policy from drifting too far into strange, reward-hacking behavior. It keeps the model's outputs fluent and coherent while still improving alignment.
A higher β means stronger regularization (stay closer to reference), while lower β allows more aggressive optimization.
Key Takeaway: The RLHF objective function balances reward maximization with a KL divergence penalty to prevent the policy from straying too far from safe, coherent behavior while still improving alignment.