The Bradley-Terry Model for Preferences
What you'll learn: How to convert human preference judgments into mathematical probabilities using reward scores.
The Core Problem
When humans prefer response A over response B, how do we model that mathematically? The Bradley-Terry model gives us an elegant solution: it assumes each response has an underlying "reward score," and the probability of preferring one response over another follows a simple formula.
The Mathematics
Imagine two model outputs have hidden quality scores: r_A for response A and r_B for response B. The Bradley-Terry model says the probability that humans prefer A over B is:
P(A > B) = exp(r_A) / (exp(r_A) + exp(r_B))
This is just the softmax function applied to two reward scores! You can rewrite it equivalently as:
P(A > B) = 1 / (1 + exp(r_B - r_A)) = sigmoid(r_A - r_B)
Why This Makes Sense
Think of reward scores like temperature readings: a small difference means preferences are uncertain (close to 50-50), while a large difference makes the better choice obvious (near 0% or 100%). The exponential function naturally captures this relationship.
Analogy: If two chess players have ratings 1500 vs 1600, the Bradley-Terry model predicts win probability based on that difference—larger gaps mean more predictable outcomes.
Connection to Reward Models
In RLHF, we train a neural network to predict these reward scores r(x) for any response x. During training, we adjust the network so that P(preferred > rejected) matches human judgments in our preference dataset.
Key Takeaway: The Bradley-Terry model transforms preference comparisons into probabilities using reward score differences, providing the mathematical foundation for training reward models in RLHF.