The Bradley-Terry Preference Model
You'll learn: How the Bradley-Terry model mathematically converts model output probabilities into preference predictions between two competing responses.
What Is the Bradley-Terry Model?
The Bradley-Terry model is a classic statistical framework for modeling pairwise comparisons. In the context of LLM alignment, it provides the mathematical foundation for predicting which of two completions a human will prefer based on learned quality scores.
Think of it like a sports ranking system: if Team A has a strength score and Team B has a strength score, the Bradley-Terry model tells you the probability that Team A beats Team B in a match.
The Core Formula
Given two model completions (let's call them "winner" and "loser") for the same prompt, the Bradley-Terry model states:
P(y_w > y_l) = exp(r(y_w)) / (exp(r(y_w)) + exp(r(y_l)))
Or equivalently, using the sigmoid function:
P(y_w > y_l) = σ(r(y_w) - r(y_l))
Where:
r(y_w)is the reward score for the preferred (winning) completionr(y_l)is the reward score for the rejected (losing) completionσis the sigmoid function:σ(x) = 1/(1 + exp(-x))
Why This Matters for RLHF
This model connects raw reward scores to actual preference probabilities. When training a reward model, we use this formula to compute the likelihood that the model's learned scores match human preferences. The bigger the difference between r(y_w) and r(y_l), the more confident we are that humans will prefer the winner.
Key Takeaway: The Bradley-Terry model transforms scalar reward scores into preference probabilities using P(win) = σ(r_winner - r_loser), providing the mathematical bridge between model outputs and human preference data.