Choosing the Judge Model
What you'll learn: How to pick the right LLM to serve as your evaluator when using LLM-as-a-Judge.
The Evaluator Must Be Stronger Than the Evaluated
When using LLM-as-a-Judge (lesson 807), one golden rule stands above all: your judge model must be more capable than the model you're evaluating. Think of it like having a teacher grade student work—you wouldn't ask a 5th grader to grade a college thesis.
Why Capability Matters
A weaker model judging a stronger model's output creates several problems:
- Blind spots: The judge may not recognize sophisticated reasoning it couldn't produce itself
- False negatives: Quality outputs might be marked as poor because the judge doesn't understand them
- Inconsistency: The judge may lack the nuance to distinguish between subtle quality differences
Practical Selection Guidelines
For most evaluation scenarios:
- If evaluating GPT-3.5-turbo: Use GPT-4 or Claude Opus as your judge
- If evaluating open-source models (Llama, Mistral): Use GPT-4, Claude Opus, or GPT-4-turbo
- If evaluating GPT-4: Consider GPT-4-turbo, Claude Opus, or ensemble judging with multiple strong models
The judge needs deep understanding of language quality, reasoning ability, and domain knowledge that exceeds what you're testing. This ensures reliable, consistent scoring across your evaluation dimensions (lesson 805).
The Cost-Quality Trade-off
Using GPT-4 or Claude Opus as judges is more expensive than weaker models, but the investment pays off in evaluation reliability. Budget-conscious teams can batch evaluations or sample test sets rather than compromise on judge quality.
Key Takeaway: Always choose a judge model that's demonstrably more capable than the model being evaluated—think expert reviewer, not peer reviewer.