Course contentsShow
← AI Engineering
Lesson 809 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Choosing the Judge Model

Selecting appropriately capable models (GPT-4, Claude) that are stronger than the model being evaluated.

Choosing the Judge Model

What you'll learn: How to pick the right LLM to serve as your evaluator when using LLM-as-a-Judge.

The Evaluator Must Be Stronger Than the Evaluated

When using LLM-as-a-Judge (lesson 807), one golden rule stands above all: your judge model must be more capable than the model you're evaluating. Think of it like having a teacher grade student work—you wouldn't ask a 5th grader to grade a college thesis.

Why Capability Matters

A weaker model judging a stronger model's output creates several problems:

  • Blind spots: The judge may not recognize sophisticated reasoning it couldn't produce itself
  • False negatives: Quality outputs might be marked as poor because the judge doesn't understand them
  • Inconsistency: The judge may lack the nuance to distinguish between subtle quality differences

Practical Selection Guidelines

For most evaluation scenarios:

  • If evaluating GPT-3.5-turbo: Use GPT-4 or Claude Opus as your judge
  • If evaluating open-source models (Llama, Mistral): Use GPT-4, Claude Opus, or GPT-4-turbo
  • If evaluating GPT-4: Consider GPT-4-turbo, Claude Opus, or ensemble judging with multiple strong models

The judge needs deep understanding of language quality, reasoning ability, and domain knowledge that exceeds what you're testing. This ensures reliable, consistent scoring across your evaluation dimensions (lesson 805).

The Cost-Quality Trade-off

Using GPT-4 or Claude Opus as judges is more expensive than weaker models, but the investment pays off in evaluation reliability. Budget-conscious teams can batch evaluations or sample test sets rather than compromise on judge quality.

Key Takeaway: Always choose a judge model that's demonstrably more capable than the model being evaluated—think expert reviewer, not peer reviewer.