Training Objective for Reward Models
What you'll learn: How to train a reward model to predict human preferences by optimizing the Bradley-Terry loss over comparison pairs.
The Core Idea
Once you've collected preference data showing which responses humans prefer (from lessons on preference data collection), you need to train your reward model to predict these preferences. The training objective is to maximize the probability that the reward model assigns a higher score to the preferred response than to the rejected one.
Bradley-Terry Loss in Action
The Bradley-Terry model gives us a clean probability framework: the probability that response A is preferred over response B is:
P(A > B) = exp(r_A) / (exp(r_A) + exp(r_B)) = sigmoid(r_A - r_B)
where r_A and r_B are the reward model's scalar outputs for each response.
For training, we minimize the negative log-likelihood of the preferred response actually being preferred:
Loss = -log(sigmoid(r_chosen - r_rejected))
Or equivalently, the cross-entropy loss where the label is always 1 (chosen is better):
Loss = -log(exp(r_chosen) / (exp(r_chosen) + exp(r_rejected)))
Why This Works
Think of it like training a judge: you show the model many pairs of responses with human verdicts, and it learns to assign scores that match those preferences. The model doesn't output "good" or "bad" labels—it outputs scalar rewards that can be compared.
The gradient naturally encourages the model to increase r_chosen and decrease r_rejected, creating a ranking that aligns with human judgment.
Key Takeaway: The Bradley-Terry loss trains reward models by minimizing the negative log-probability of the chosen response having a higher reward than the rejected one, turning preference pairs into a differentiable ranking objective.