Judge Prompt Design Fundamentals
What you'll learn: How to craft effective prompts that turn an LLM into a reliable evaluator by specifying clear criteria, evaluation dimensions, and scoring rubrics.
The Core Idea
When using an LLM as a judge (as introduced in lesson 3161), the quality of evaluation depends entirely on your judge prompt. Unlike calling a fixed metric function, you're instructing a model to reason about quality—so your prompt becomes your "evaluation specification."
Think of it like writing instructions for a human grader: vague instructions ("Is this good?") produce inconsistent results, while detailed rubrics with clear criteria yield reliable, reproducible scores.
Essential Components of Judge Prompts
Criteria Specification
Define what aspects matter. For a summarization task: factual accuracy, brevity, coherence. For code generation: correctness, efficiency, readability. List these explicitly so the judge knows what to evaluate.
Evaluation Dimensions
Break complex judgments into separate dimensions. Instead of "overall quality," specify dimensions like "relevance (1-5)," "fluency (1-5)," "safety (pass/fail)." This reduces conflation and improves interpretability.
Scoring Rubrics
Provide anchor points for each score level. For example:
- Score 5: Completely addresses the query with no errors
- Score 3: Partially addresses query but contains minor inaccuracies
- Score 1: Fails to address query or contains major errors
Without rubrics, different judge runs may interpret "good" differently, causing unreliable variance.
Design Pattern
Structure your judge prompt: context → task description → criteria → rubric → output format. Be explicit about edge cases (ties, insufficient information). Test consistency by running the same judgment multiple times.
Key Takeaway: Effective judge prompts specify what to evaluate (criteria), how finely to evaluate it (dimensions), and what each score means (rubrics)—turning subjective judgment into structured, reproducible evaluation.