Inter-Annotator Agreement Metrics
What you'll learn: How to quantify whether your human evaluators agree with each other using Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha.
Why Agreement Matters
When multiple annotators label the same data (as you learned in designing annotation guidelines), they won't always agree. But how much disagreement is acceptable? Inter-annotator agreement (IAA) metrics tell you whether your evaluators are consistently interpreting your task, or if your guidelines need refinement.
Think of it like judges scoring figure skating—if all judges give wildly different scores for the same performance, something's wrong with the scoring system itself.
The Three Key Metrics
Cohen's Kappa
Measures agreement between two annotators on categorical labels, correcting for chance agreement. Values range from -1 to 1:
- 1.0 = perfect agreement
- 0.0 = agreement no better than random chance
- < 0 = systematic disagreement
It's your go-to for binary or multi-class tasks with exactly two raters.
Fleiss' Kappa
Extends Cohen's kappa to three or more annotators rating the same items. Unlike averaging pairwise kappas, Fleiss' kappa treats all raters as a single pool. Use this when you have multiple annotators per example but not necessarily the same annotators for all examples.
Krippendorff's Alpha
The most flexible metric—handles any number of annotators, any data type (nominal, ordinal, interval, ratio), and missing data. It's considered the gold standard for complex annotation projects, especially when annotators don't label all items equally.
Interpreting Values
A common rule of thumb:
- > 0.8: substantial agreement
- 0.6–0.8: moderate agreement
- < 0.6: questionable—revise guidelines or retrain annotators
Key Takeaway: Inter-annotator agreement metrics like Cohen's kappa (2 raters), Fleiss' kappa (3+ raters), and Krippendorff's alpha (flexible, any scale) quantify labeling consistency and reveal whether your evaluation task is well-defined or needs refinement.