Course contentsShow
← Machine Learning and Deep Learning
Lesson 1782 of 3,53838. Instruction Tuning and AlignmentFree lesson

Training Objective for Reward Models

Optimizing the Bradley-Terry loss to predict which response humans prefer in each comparison pair.

Training Objective for Reward Models

What you'll learn: How to train a reward model to predict human preferences by optimizing the Bradley-Terry loss over comparison pairs.

The Core Idea

Once you've collected preference data showing which responses humans prefer (from lessons on preference data collection), you need to train your reward model to predict these preferences. The training objective is to maximize the probability that the reward model assigns a higher score to the preferred response than to the rejected one.

Bradley-Terry Loss in Action

The Bradley-Terry model gives us a clean probability framework: the probability that response A is preferred over response B is:

P(A > B) = exp(r_A) / (exp(r_A) + exp(r_B)) = sigmoid(r_A - r_B)

where r_A and r_B are the reward model's scalar outputs for each response.

For training, we minimize the negative log-likelihood of the preferred response actually being preferred:

Loss = -log(sigmoid(r_chosen - r_rejected))

Or equivalently, the cross-entropy loss where the label is always 1 (chosen is better):

Loss = -log(exp(r_chosen) / (exp(r_chosen) + exp(r_rejected)))

Why This Works

Think of it like training a judge: you show the model many pairs of responses with human verdicts, and it learns to assign scores that match those preferences. The model doesn't output "good" or "bad" labels—it outputs scalar rewards that can be compared.

The gradient naturally encourages the model to increase r_chosen and decrease r_rejected, creating a ranking that aligns with human judgment.

Key Takeaway: The Bradley-Terry loss trains reward models by minimizing the negative log-probability of the chosen response having a higher reward than the rejected one, turning preference pairs into a differentiable ranking objective.