Rubrics and Scoring Criteria
What you'll learn: How to provide explicit scoring rubrics in your judge prompts to ensure LLM evaluators produce consistent, interpretable scores.
Why Explicit Rubrics Matter
When using an LLM as a judge, vague instructions like "rate this response" produce inconsistent, subjective scores. One evaluation might give a 7/10, another a 9/10 for identical outputs—with no clear reasoning behind either.
Explicit rubrics solve this by defining exactly what each score means. Think of it like a teacher's grading rubric: instead of "good essay gets an A," you specify "an A requires a clear thesis, three supporting arguments with evidence, and proper citations."
Anatomy of a Good Rubric
A scoring rubric should include:
- Score range: Define your scale (1-5, 0-10, letter grades)
- Criteria per level: Explain what distinguishes each score
- Observable behaviors: Use concrete, measurable qualities
- Examples (when possible): Show what a "3" vs "5" looks like
For example, when evaluating response helpfulness:
Score 5: Directly answers the question with accurate details and actionable steps
Score 4: Answers the question but lacks some detail or clarity
Score 3: Partially answers but misses key elements
Score 2: Vaguely related but doesn't address the core question
Score 1: Off-topic or unhelpful
Benefits of Structured Rubrics
Rubrics make your evaluation framework:
- Reproducible: Different judge models (or the same model at different times) produce similar scores
- Interpretable: Stakeholders understand why something scored 3 vs 5
- Debuggable: You can identify whether low scores reflect actual quality issues or rubric problems
Key Takeaway: Explicit rubrics transform LLM-as-a-judge from subjective opinion into structured, consistent evaluation by defining exactly what each score represents with observable criteria.