Bradley-Terry Model for Preferences
You'll learn: How to convert human preference comparisons into numerical reward signals using a probabilistic model based on logistic regression.
The Core Idea
When humans compare two model outputs and say "A is better than B," how do you turn that judgment into a number the reward model can learn? The Bradley-Terry model provides the mathematical framework.
The model assumes each output has an underlying "quality" or reward score. If output A has reward r_A and output B has reward r_B, the probability that humans prefer A over B follows a logistic function:
P(A ≻ B) = 1 / (1 + exp(r_B - r_A))
This is equivalent to:
P(A ≻ B) = exp(r_A) / (exp(r_A) + exp(r_B))
Why This Works
Think of it like a sports match: if team A has strength r_A and team B has strength r_B, the Bradley-Terry model predicts the probability A wins based on their relative strengths. The bigger the gap between r_A and r_B, the more confident the prediction.
In RLHF, your reward model outputs these r scores. During training, you maximize the likelihood that the model assigns higher probability to the human-preferred completions. The loss function becomes:
Loss = -log(P(preferred ≻ rejected))
= -log(exp(r_preferred) / (exp(r_preferred) + exp(r_rejected)))
This is exactly binary cross-entropy on the preference pairs!
Key Takeaway: The Bradley-Terry model transforms pairwise human preferences into a logistic regression problem, letting you train a reward model that assigns scalar scores matching human judgments of quality.