Course contentsShow
← AI Engineering
Lesson 848 of 1,88621. Human Evaluation and FeedbackFree lesson

Iterating on Rubrics with Data

Analyzing annotation patterns to identify unclear criteria and refine rubrics over time.

Iterating on Rubrics with Data

What you'll learn: How to use annotation patterns to spot unclear rubric criteria and systematically improve your evaluation guidelines.

The Core Idea

Your first evaluation rubric is never perfect. As human annotators work through rating LLM outputs, patterns emerge in their behavior—clusters of disagreement, confusing edge cases, or criteria that people interpret differently. By analyzing this annotation data, you can identify which parts of your rubric need clarification and iteratively refine them.

Think of it like refining a recipe: the first version might say "add salt to taste," but after watching ten cooks disagree wildly on amounts, you'd revise it to "add 1 teaspoon of salt." Your rubric evolves the same way.

What to Look For

High disagreement areas: When inter-annotator agreement is low on specific criteria, that criterion is probably ambiguous. For example, if annotators split 50/50 on rating "helpfulness," your definition of helpful needs more concrete examples.

Edge case clusters: If annotators frequently flag the same types of outputs as confusing, add explicit guidance for those scenarios to your rubric.

Scoring pattern analysis: If everyone uses only the extreme ends of your 5-point scale (all 1s or 5s), your middle values might lack clear definitions.

The Iteration Process

  1. Collect annotations with your current rubric
  2. Calculate inter-annotator agreement per criterion
  3. Review disagreement cases with annotators
  4. Identify ambiguous language or missing examples
  5. Update rubric with clearer definitions and examples
  6. Re-calibrate annotators on the revised rubric
  7. Measure whether agreement improves

This cycle transforms your rubric from a rough sketch into a reliable measurement tool.

Key Takeaway: Treat your evaluation rubric as a living document—analyze annotation patterns to find confusion points, then refine criteria with concrete examples and clearer definitions based on real disagreement data.