Course contentsShow
← Machine Learning and Deep Learning
Lesson 3163 of 3,53869. LLM EvaluationFree lesson

Judge Prompt Design Fundamentals

Crafting effective judge prompts: criteria specification, evaluation dimensions, and scoring rubrics.

Judge Prompt Design Fundamentals

What you'll learn: How to craft effective prompts that turn an LLM into a reliable evaluator by specifying clear criteria, evaluation dimensions, and scoring rubrics.

The Core Idea

When using an LLM as a judge (as introduced in lesson 3161), the quality of evaluation depends entirely on your judge prompt. Unlike calling a fixed metric function, you're instructing a model to reason about quality—so your prompt becomes your "evaluation specification."

Think of it like writing instructions for a human grader: vague instructions ("Is this good?") produce inconsistent results, while detailed rubrics with clear criteria yield reliable, reproducible scores.

Essential Components of Judge Prompts

Criteria Specification

Define what aspects matter. For a summarization task: factual accuracy, brevity, coherence. For code generation: correctness, efficiency, readability. List these explicitly so the judge knows what to evaluate.

Evaluation Dimensions

Break complex judgments into separate dimensions. Instead of "overall quality," specify dimensions like "relevance (1-5)," "fluency (1-5)," "safety (pass/fail)." This reduces conflation and improves interpretability.

Scoring Rubrics

Provide anchor points for each score level. For example:

  • Score 5: Completely addresses the query with no errors
  • Score 3: Partially addresses query but contains minor inaccuracies
  • Score 1: Fails to address query or contains major errors

Without rubrics, different judge runs may interpret "good" differently, causing unreliable variance.

Design Pattern

Structure your judge prompt: context → task description → criteria → rubric → output format. Be explicit about edge cases (ties, insufficient information). Test consistency by running the same judgment multiple times.

Key Takeaway: Effective judge prompts specify what to evaluate (criteria), how finely to evaluate it (dimensions), and what each score means (rubrics)—turning subjective judgment into structured, reproducible evaluation.