Course contentsShow
← AI Engineering
Lesson 826 of 1,88620. Evaluation Frameworks for LLM SystemsFree lesson

Inter-Annotator Agreement

Measuring consistency between multiple annotators using metrics like Cohen's kappa to validate ground truth quality.

Inter-Annotator Agreement

You'll learn how to measure consistency between human annotators to validate your ground truth dataset quality.

What Is Inter-Annotator Agreement?

When you create ground truth data through manual annotation (as covered in Lesson 821), you need to ensure your annotators agree with each other. Inter-annotator agreement measures how consistently multiple humans label the same data. If annotators disagree wildly, your "ground truth" isn't actually ground truth—it's subjective noise.

Think of it like this: if three judges at a competition can't agree on who won, the scoring system itself is broken. Similarly, if your annotators can't agree on whether a response is "good" or "bad," your evaluation framework has a foundational problem.

Cohen's Kappa: The Gold Standard

The most common metric is Cohen's kappa (κ), which measures agreement between two annotators while accounting for chance agreement. It ranges from -1 to 1:

  • κ = 1: Perfect agreement
  • κ = 0: Agreement no better than random chance
  • κ < 0: Agreement worse than chance (systematic disagreement)

Generally, κ > 0.6 is acceptable, κ > 0.8 is strong agreement.

Why This Matters

Before you invest in a large annotation project, run a pilot with 50-100 examples annotated by all team members. Calculate inter-annotator agreement. Low agreement means you need clearer guidelines, better training, or simpler annotation tasks. High agreement validates that your annotation workflow produces reliable ground truth for evaluating your LLM system.

Key Takeaway: Inter-annotator agreement quantifies whether your human annotators are consistent enough to produce reliable ground truth—use metrics like Cohen's kappa to validate annotation quality before scaling.