Course contentsShow
← Machine Learning and Deep Learning
Lesson 3175 of 3,53869. LLM EvaluationFree lesson

Elo Rating Systems for LLMs

Applying Elo ratings from game theory to rank LLMs based on head-to-head win rates across many comparisons.

Elo Rating Systems for LLMs

You'll learn how to borrow chess's Elo rating system to create a unified ranking of large language models based on thousands of head-to-head matchups.

The Chess Analogy

In chess, players earn Elo ratings that rise when they beat strong opponents and fall when they lose to weaker ones. The system expects higher-rated players to win more often, and adjusts ratings accordingly after each match. We apply this exact logic to LLMs: instead of chess games, we run pairwise comparisons where human judges (or LLM judges) pick which of two models gave the better response.

How Elo Works for LLMs

Start by assigning all models an initial rating (typically 1000 or 1500). For each pairwise comparison:

  1. Calculate expected win probability using the rating difference (a 400-point gap means ~10× higher odds)
  2. Record the actual outcome (Model A wins, loses, or ties)
  3. Update both ratings based on whether the result was surprising or expected

A weak model beating GPT-4 would cause a massive rating swing, while GPT-4 beating the weak model barely changes anything—this captures information efficiently.

Why This Beats Simple Win Rates

Raw win rates don't account for opponent strength. A model with 70% wins against weak baselines isn't necessarily better than one with 60% wins against top-tier competitors. Elo naturally weights victories by opponent quality, producing a single meaningful leaderboard from messy head-to-head data collected across different prompts, domains, and comparison batches.

Key Takeaway: Elo ratings transform thousands of pairwise LLM comparisons into a unified ranking that accounts for opponent strength, making them ideal for aggregating results from human or LLM-judge evaluations at scale.