Course contentsShow
← Machine Learning and Deep Learning
Lesson 1805 of 3,53838. Instruction Tuning and AlignmentFree lesson

The Bradley-Terry Preference Model

Mathematical foundation relating model probabilities to human preferences between completions.

The Bradley-Terry Preference Model

You'll learn: How the Bradley-Terry model mathematically converts model output probabilities into preference predictions between two competing responses.

What Is the Bradley-Terry Model?

The Bradley-Terry model is a classic statistical framework for modeling pairwise comparisons. In the context of LLM alignment, it provides the mathematical foundation for predicting which of two completions a human will prefer based on learned quality scores.

Think of it like a sports ranking system: if Team A has a strength score and Team B has a strength score, the Bradley-Terry model tells you the probability that Team A beats Team B in a match.

The Core Formula

Given two model completions (let's call them "winner" and "loser") for the same prompt, the Bradley-Terry model states:

P(y_w > y_l) = exp(r(y_w)) / (exp(r(y_w)) + exp(r(y_l)))

Or equivalently, using the sigmoid function:

P(y_w > y_l) = σ(r(y_w) - r(y_l))

Where:

  • r(y_w) is the reward score for the preferred (winning) completion
  • r(y_l) is the reward score for the rejected (losing) completion
  • σ is the sigmoid function: σ(x) = 1/(1 + exp(-x))

Why This Matters for RLHF

This model connects raw reward scores to actual preference probabilities. When training a reward model, we use this formula to compute the likelihood that the model's learned scores match human preferences. The bigger the difference between r(y_w) and r(y_l), the more confident we are that humans will prefer the winner.

Key Takeaway: The Bradley-Terry model transforms scalar reward scores into preference probabilities using P(win) = σ(r_winner - r_loser), providing the mathematical bridge between model outputs and human preference data.