Course contentsShow
← Machine Learning and Deep Learning
Lesson 1795 of 3,53838. Instruction Tuning and AlignmentFree lesson

Value Function Learning in RLHF

Training a critic network to estimate expected rewards for variance reduction in policy gradients.

Value Function Learning in RLHF

You'll learn how training a critic network to estimate expected rewards helps stabilize policy gradient updates in RLHF.

The Core Idea

When fine-tuning language models with PPO, we need to compute advantages—the difference between actual rewards and expected rewards. But raw reward signals from individual completions are noisy and high-variance. To reduce this variance, we train a value function (also called a critic network) that predicts "how good is this state?" for any partial text sequence.

Think of the value function as an experienced coach watching your policy play. Instead of waiting to see the final score (reward) after every game, the coach can estimate mid-game: "Given where you are now, you'll probably end up with a score around X." This prediction helps you understand whether a specific move was better or worse than typical.

How It Works

The value function is typically a separate neural network head (or copy) that takes the same hidden states as your policy model and outputs a single scalar: the estimated expected return from that point forward. During training:

  1. The policy generates completions and receives rewards from the reward model
  2. The value function predicts what it thinks the reward will be
  3. We compute the prediction error (actual reward - predicted value)
  4. This error trains both the advantage estimates (for policy updates) and the value function itself (via supervised learning)

By subtracting the value baseline from actual rewards, we get lower-variance advantage estimates, making policy gradient updates more stable and sample-efficient.

Key Takeaway: The value function is a learned baseline that predicts expected rewards, reducing variance in advantage estimation and enabling more stable PPO training for language model alignment.