Course contentsShow
← Machine Learning and Deep Learning
Lesson 464 of 3,53811. Model Evaluation MetricsFree lesson

Cohen's Kappa: Agreement Beyond Chance

Measuring classification performance while accounting for random chance agreement.

Cohen's Kappa: Agreement Beyond Chance

You'll learn how to measure classifier agreement while correcting for lucky guesses.

What's Wrong with Raw Accuracy?

Imagine a disease that affects only 5% of people. A lazy classifier that always predicts "healthy" achieves 95% accuracy by pure chance—no learning required! Raw accuracy can be misleading when random guessing produces good-looking results.

The Core Idea

Cohen's Kappa measures how much better your classifier performs compared to random chance. It asks: "If I randomly assigned predictions in proportion to the class frequencies, how often would I be right by accident? And how much better than that is my actual model?"

The formula adjusts your confusion matrix-based accuracy by subtracting expected agreement:

Kappa = (Observed Accuracy - Expected Accuracy) / (1 - Expected Accuracy)
  • Observed Accuracy: Your model's actual accuracy (from the confusion matrix)
  • Expected Accuracy: The accuracy a random classifier would achieve given the class distributions

Interpreting Kappa Values

  • κ = 1: Perfect agreement (all predictions correct)
  • κ = 0: Agreement no better than random chance
  • κ < 0: Worse than random guessing (rare, indicates problems)
  • κ = 0.6-0.8: Substantial agreement
  • κ > 0.8: Almost perfect agreement

Why It Matters

Cohen's Kappa is especially valuable for imbalanced datasets where accuracy alone misleads. It ensures you're evaluating genuine learning, not lucky baseline performance.

Key Takeaway: Cohen's Kappa corrects accuracy for chance agreement, giving you a more honest assessment of classifier performance, especially on imbalanced data.