Course contentsShow
← Machine Learning and Deep Learning
Lesson 3173 of 3,53869. LLM EvaluationFree lesson

Introduction to Win Rate Metrics

Understanding win rate as a binary preference metric where humans choose between two model outputs.

Introduction to Win Rate Metrics

You'll learn: How win rate provides a simple, intuitive way to compare LLM outputs by counting which model humans prefer more often.

The Core Idea

After you've built an LLM judge or gathered human evaluations (as you've learned in recent lessons), you need a single number to summarize performance. Win rate is the simplest preference metric: given two model outputs for the same input, a human (or LLM judge) picks the better one. The win rate is just the percentage of times your model wins these head-to-head comparisons.

Think of it like a sports tournament: if Model A beats Model B in 65 out of 100 matches, Model A has a 65% win rate against Model B.

Why Win Rate Matters

Unlike absolute scoring systems where you rate outputs on a scale (1-5 stars), binary preference is often easier and more reliable for evaluators. It's simpler to decide "which answer is better?" than "is this a 3.5 or 4.0?"

Win rate directly answers practical questions:

  • "Should we deploy the new fine-tuned version?"
  • "Which prompt template performs better?"
  • "Does model X outperform our baseline?"

This metric naturally extends the pairwise comparison approach you learned about with LLM judges. Instead of analyzing individual judgments, win rate aggregates them into an interpretable percentage.

Computing Win Rate

Win Rate = (Number of Wins) / (Total Comparisons)

If there are ties, you typically count them as 0.5 wins each, or exclude them entirely depending on your evaluation protocol.

Key Takeaway: Win rate transforms pairwise preference judgments into a single percentage metric, making it easy to compare models and track improvements across iterations.