Course contentsShow
← Machine Learning and Deep Learning
Lesson 3110 of 3,53868. Evaluation in PracticeFree lesson

Inter-Annotator Agreement Metrics

Measuring consistency with Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha for reliability assessment.

Inter-Annotator Agreement Metrics

What you'll learn: How to quantify whether your human evaluators agree with each other using Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha.

Why Agreement Matters

When multiple annotators label the same data (as you learned in designing annotation guidelines), they won't always agree. But how much disagreement is acceptable? Inter-annotator agreement (IAA) metrics tell you whether your evaluators are consistently interpreting your task, or if your guidelines need refinement.

Think of it like judges scoring figure skating—if all judges give wildly different scores for the same performance, something's wrong with the scoring system itself.

The Three Key Metrics

Cohen's Kappa

Measures agreement between two annotators on categorical labels, correcting for chance agreement. Values range from -1 to 1:

  • 1.0 = perfect agreement
  • 0.0 = agreement no better than random chance
  • < 0 = systematic disagreement

It's your go-to for binary or multi-class tasks with exactly two raters.

Fleiss' Kappa

Extends Cohen's kappa to three or more annotators rating the same items. Unlike averaging pairwise kappas, Fleiss' kappa treats all raters as a single pool. Use this when you have multiple annotators per example but not necessarily the same annotators for all examples.

Krippendorff's Alpha

The most flexible metric—handles any number of annotators, any data type (nominal, ordinal, interval, ratio), and missing data. It's considered the gold standard for complex annotation projects, especially when annotators don't label all items equally.

Interpreting Values

A common rule of thumb:

  • > 0.8: substantial agreement
  • 0.6–0.8: moderate agreement
  • < 0.6: questionable—revise guidelines or retrain annotators

Key Takeaway: Inter-annotator agreement metrics like Cohen's kappa (2 raters), Fleiss' kappa (3+ raters), and Krippendorff's alpha (flexible, any scale) quantify labeling consistency and reveal whether your evaluation task is well-defined or needs refinement.