Inter-Annotator Agreement Metrics
What you'll learn: How to measure and improve the consistency and reliability of your fine-tuning data through statistical agreement metrics.
Why Agreement Matters
When multiple annotators label training data, they won't always agree. If two experienced annotators give completely different labels to the same example, something's wrong—either your guidelines are unclear, the task is ambiguous, or the annotators need better training. Inter-annotator agreement (IAA) metrics quantify this consistency numerically, helping you catch data quality issues before they poison your fine-tuned model.
Cohen's Kappa: Beyond Simple Agreement
The most widely used metric is Cohen's kappa (κ). Unlike raw percentage agreement (which can be misleadingly high), kappa accounts for chance agreement.
The intuition: If you have two categories and annotators randomly guess, they'll agree 50% of the time by pure luck. Kappa adjusts for this baseline:
- κ = 1.0: Perfect agreement
- κ = 0.0: Agreement no better than random chance
- κ < 0.0: Worse than random (systematic disagreement)
Rule of thumb: κ > 0.8 indicates strong agreement; κ = 0.6–0.8 is moderate; below 0.6 suggests your annotation process needs work.
Other Useful Metrics
- Fleiss' kappa: Extends Cohen's kappa to three or more annotators
- Krippendorff's alpha: Handles missing data and different measurement levels
- Percentage agreement: Simple but useful as a quick sanity check
Practical Workflow
- Have multiple annotators label the same small sample (~50-100 examples)
- Calculate IAA metrics
- Review disagreements to identify confusing cases
- Refine annotation guidelines based on patterns
- Re-train annotators and measure again
Key Takeaway: Cohen's kappa and related IAA metrics help you catch inconsistent annotations early, ensuring your fine-tuning data is reliable—a model trained on noisy labels will produce noisy outputs.