Course contentsShow
← Machine Learning and Deep Learning
Lesson 3178 of 3,53869. LLM EvaluationFree lesson

Annotation Quality and Inter-Rater Agreement

Measuring annotator consistency with Cohen's kappa and ensuring reliable human preference judgments.

Annotation Quality and Inter-Rater Agreement

You'll learn how to measure whether human annotators consistently agree on LLM outputs, ensuring your evaluation data is reliable.

The Core Challenge

When evaluating LLMs through human judgments (like pairwise comparisons or quality ratings), you need multiple annotators to avoid individual bias. But if annotators disagree wildly, your evaluation metrics become meaningless noise. Inter-rater agreement quantifies how consistently different humans make the same judgments on identical examples.

Cohen's Kappa: Beyond Simple Agreement

Imagine two annotators labeling 100 LLM responses as "good" or "bad." They agree on 80 cases—seems great! But if 70% of responses are naturally "good," random chance would produce ~58% agreement. Cohen's kappa corrects for this:

κ = (observed agreement - expected agreement) / (1 - expected agreement)

  • κ = 1.0: Perfect agreement
  • κ = 0.0: Agreement by chance alone
  • κ < 0.6: Generally considered questionable reliability
  • κ > 0.8: Strong agreement

Why This Matters for LLM Evaluation

In systems like Chatbot Arena or when collecting preference data for RLHF, you're building datasets where annotators choose between model outputs. Low agreement means:

  • Your training signal for alignment is noisy
  • Win rates become unreliable
  • Benchmark rankings may reflect annotation variance, not model quality

Best practices: Run pilot annotations, measure kappa early, refine guidelines when agreement is low, and use gold standard examples to maintain consistency across annotators.

Key Takeaway: Cohen's kappa separates genuine annotator consensus from chance agreement, helping you build reliable human evaluation datasets—essential when human preference judgments drive your LLM's development and ranking.