Cohen's Kappa: Agreement Beyond Chance
You'll learn how to measure classifier agreement while correcting for lucky guesses.
What's Wrong with Raw Accuracy?
Imagine a disease that affects only 5% of people. A lazy classifier that always predicts "healthy" achieves 95% accuracy by pure chance—no learning required! Raw accuracy can be misleading when random guessing produces good-looking results.
The Core Idea
Cohen's Kappa measures how much better your classifier performs compared to random chance. It asks: "If I randomly assigned predictions in proportion to the class frequencies, how often would I be right by accident? And how much better than that is my actual model?"
The formula adjusts your confusion matrix-based accuracy by subtracting expected agreement:
Kappa = (Observed Accuracy - Expected Accuracy) / (1 - Expected Accuracy)
- Observed Accuracy: Your model's actual accuracy (from the confusion matrix)
- Expected Accuracy: The accuracy a random classifier would achieve given the class distributions
Interpreting Kappa Values
- κ = 1: Perfect agreement (all predictions correct)
- κ = 0: Agreement no better than random chance
- κ < 0: Worse than random guessing (rare, indicates problems)
- κ = 0.6-0.8: Substantial agreement
- κ > 0.8: Almost perfect agreement
Why It Matters
Cohen's Kappa is especially valuable for imbalanced datasets where accuracy alone misleads. It ensures you're evaluating genuine learning, not lucky baseline performance.
Key Takeaway: Cohen's Kappa corrects accuracy for chance agreement, giving you a more honest assessment of classifier performance, especially on imbalanced data.