Bradley-Terry Model for Rankings
What you'll learn: How the Bradley-Terry statistical model transforms pairwise win/loss comparisons into a global ranking of LLM capabilities.
The Core Idea
After running many pairwise comparisons where human judges or LLM-judges pick winners between model outputs, you face a question: how do you aggregate thousands of individual matchups into a single, coherent ranking?
The Bradley-Terry model provides an elegant answer. It assumes each model has a latent "strength" parameter, and the probability that model i beats model j follows a simple logistic form:
P(i beats j) = strength_i / (strength_i + strength_j)
Given your collection of pairwise outcomes (Model A beat Model B 73 times out of 100, etc.), you use maximum likelihood estimation to solve for the strength parameters that best explain your observed win rates. These strengths directly translate into a global ranking.
Why This Matters
Unlike Elo ratings (which update sequentially based on match history), Bradley-Terry computes strengths simultaneously from all observed comparisons. This makes it ideal when you have a complete or near-complete matrix of pairwise results—common in controlled LLM evaluation where you can test all model pairs systematically.
The model handles intransitive outcomes gracefully (where A beats B, B beats C, but C beats A) by finding the maximum-likelihood strengths that best reconcile contradictory signals.
Connection to Win Rates
Your pairwise win rates from earlier lessons become the input data. The Bradley-Terry model converts those percentages into interpretable strength scores that preserve relative ordering and satisfy statistical properties.
Key Takeaway: The Bradley-Terry model transforms raw pairwise comparison data into statistically-grounded global rankings by estimating latent strength parameters that maximize the likelihood of observed match outcomes.