Pairwise Comparison vs Absolute Scoring
What you'll learn: Two fundamental approaches to having LLMs judge quality—comparing two outputs side-by-side versus rating a single output on its own merits.
The Core Distinction
When using LLM-as-Judge (where one LLM evaluates another's outputs), you face a strategic choice in how to structure the evaluation task:
Pairwise Comparison presents the judge model with two candidate outputs (e.g., from Model A and Model B) for the same input, asking "Which is better?" The judge picks a winner or declares a tie.
Absolute Scoring shows the judge a single output in isolation, asking it to rate quality on a numeric scale (1-5 stars, 0-100 points) or categorical labels (poor/good/excellent) without seeing alternatives.
Why the Paradigm Matters
Pairwise comparison mirrors how humans naturally judge quality—it's often easier to say "this is better than that" than to assign an objective score. This approach reduces position bias (the judge favoring the first or second option based on order) when properly randomized, and produces clear relative rankings. However, it requires N×(N-1)/2 comparisons to rank N systems, making it computationally expensive.
Absolute scoring is efficient—one evaluation per output—and provides interpretable numeric feedback. But LLMs often exhibit scale inconsistency (what's a "4" versus "5"?), positivity bias (avoiding low scores), and anchor effects (rating everything near the scale's middle).
Many practitioners use pairwise for final model selection and absolute scoring for large-scale monitoring, combining the reliability of comparison with the efficiency of independent rating.
Key Takeaway: Pairwise comparison trades computational cost for more reliable relative judgments, while absolute scoring offers efficiency at the risk of scale inconsistency—choose based on whether you need rankings or standalone quality estimates.