Course contentsShow
← Machine Learning and Deep Learning
Lesson 1771 of 3,53838. Instruction Tuning and AlignmentFree lesson

The RLHF Objective Function

Combining reward maximization with KL divergence penalty to prevent the policy from drifting too far from reference.

The RLHF Objective Function

What you'll learn: How RLHF balances reward maximization with staying close to a reference model to create stable, aligned language models.

The Core Challenge

After training a reward model, we want to use reinforcement learning to optimize our policy model to maximize predicted rewards. But here's the problem: if we only maximize reward, the policy might exploit quirks in the reward model, generating outputs that score high but are nonsensical or harmful. It's like a student who figures out how to game a test without actually learning the material.

The Solution: A Balanced Objective

The RLHF objective function combines two competing goals:

  1. Maximize expected reward from the reward model
  2. Stay close to the reference model (usually the SFT model)

Mathematically, the objective looks like:

objective = E[reward(x, y)] - β * KL(π_θ || π_ref)

Where:

  • reward(x, y) is the reward model's score for prompt x and response y
  • KL(π_θ || π_ref) is the KL divergence measuring how much the policy π_θ differs from the reference π_ref
  • β is a hyperparameter controlling the trade-off

Why KL Divergence Matters

Think of the reference model as a safety anchor. The KL divergence penalty prevents the policy from drifting too far into strange, reward-hacking behavior. It keeps the model's outputs fluent and coherent while still improving alignment.

A higher β means stronger regularization (stay closer to reference), while lower β allows more aggressive optimization.

Key Takeaway: The RLHF objective function balances reward maximization with a KL divergence penalty to prevent the policy from straying too far from safe, coherent behavior while still improving alignment.