Course contentsShow
← AI Engineering
Lesson 811 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Rubrics and Scoring Criteria

Providing explicit rubrics in judge prompts to ensure consistent, interpretable scoring.

Rubrics and Scoring Criteria

What you'll learn: How to provide explicit scoring rubrics in your judge prompts to ensure LLM evaluators produce consistent, interpretable scores.

Why Explicit Rubrics Matter

When using an LLM as a judge, vague instructions like "rate this response" produce inconsistent, subjective scores. One evaluation might give a 7/10, another a 9/10 for identical outputs—with no clear reasoning behind either.

Explicit rubrics solve this by defining exactly what each score means. Think of it like a teacher's grading rubric: instead of "good essay gets an A," you specify "an A requires a clear thesis, three supporting arguments with evidence, and proper citations."

Anatomy of a Good Rubric

A scoring rubric should include:

  1. Score range: Define your scale (1-5, 0-10, letter grades)
  2. Criteria per level: Explain what distinguishes each score
  3. Observable behaviors: Use concrete, measurable qualities
  4. Examples (when possible): Show what a "3" vs "5" looks like

For example, when evaluating response helpfulness:

Score 5: Directly answers the question with accurate details and actionable steps
Score 4: Answers the question but lacks some detail or clarity
Score 3: Partially answers but misses key elements
Score 2: Vaguely related but doesn't address the core question
Score 1: Off-topic or unhelpful

Benefits of Structured Rubrics

Rubrics make your evaluation framework:

  • Reproducible: Different judge models (or the same model at different times) produce similar scores
  • Interpretable: Stakeholders understand why something scored 3 vs 5
  • Debuggable: You can identify whether low scores reflect actual quality issues or rubric problems

Key Takeaway: Explicit rubrics transform LLM-as-a-judge from subjective opinion into structured, consistent evaluation by defining exactly what each score represents with observable criteria.