Course contentsShow
← AI Engineering
Lesson 842 of 1,88621. Human Evaluation and FeedbackFree lesson

Inter-Annotator Agreement

Measuring consistency between annotators using Cohen's kappa and Fleiss' kappa metrics.

Inter-Annotator Agreement

What you'll learn: How to measure whether different human evaluators are judging your LLM outputs consistently using statistical metrics.

Why Agreement Matters

When you've designed evaluation rubrics and rating scales (from the previous lessons), you need multiple annotators to label your data. But how do you know they're interpreting the rubric the same way? If one annotator thinks a response is "excellent" while another calls the same response "poor," your ground truth becomes unreliable.

Inter-annotator agreement (IAA) measures the consistency between different human judges. High agreement means your rubric is clear and your evaluators are aligned. Low agreement signals ambiguous criteria or insufficient annotator training.

Cohen's Kappa: Two Annotators

When you have two annotators, use Cohen's kappa (κ). It measures agreement beyond what you'd expect by random chance.

  • κ = 1.0: Perfect agreement
  • κ = 0.0: Agreement is no better than random chance
  • κ < 0.0: Agreement is worse than random (annotators systematically disagree)

Rule of thumb:

  • κ > 0.8: Excellent agreement
  • 0.6 - 0.8: Substantial agreement
  • 0.4 - 0.6: Moderate agreement
  • < 0.4: Poor agreement (revise rubric or retrain annotators)

Fleiss' Kappa: Three or More Annotators

When you have three or more annotators, use Fleiss' kappa. It generalizes Cohen's kappa to multiple raters judging the same items. The interpretation scale is identical.

Key Takeaway

Measure inter-annotator agreement with Cohen's kappa (2 annotators) or Fleiss' kappa (3+) to validate that your evaluation rubric is clear and consistently applied—low agreement (<0.6) means you need to refine your rubric or retrain your evaluators before trusting the ground truth labels.