Confusion Matrix Disparities
What you'll learn: How different demographic groups can experience dramatically different types of prediction errors, even when overall accuracy looks fair.
The Hidden Inequality in Error Types
When evaluating fairness, overall accuracy can mask critical disparities. A model might achieve 85% accuracy for both Group A and Group B, yet treat them very differently in how it makes mistakes.
The confusion matrix breaks predictions into four categories: True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). Confusion matrix disparities occur when error rates derived from these cells differ significantly across demographic groups.
Why This Matters
Consider a loan approval model:
- Group A might face a high False Positive Rate (wrongly denied loans they could repay)
- Group B might face a high False Negative Rate (wrongly approved for loans they'll default on)
Both harm people differently. High FPR denies opportunities; high FNR leads to debt traps.
Key Metrics to Compare
When analyzing confusion matrix disparities, compute these rates separately for each group:
- False Positive Rate (FPR): FP / (FP + TN) — how often negatives are misclassified
- False Negative Rate (FNR): FN / (FN + TP) — how often positives are missed
- Precision: TP / (TP + FP) — accuracy of positive predictions
- Recall: TP / (TP + FN) — coverage of actual positives
Large gaps in these metrics between groups signal that your model distributes harm unevenly, even if aggregate performance seems balanced.
The Insight
A "fair" model by one metric (say, Demographic Parity) can still exhibit severe confusion matrix disparities. A group might receive equal positive prediction rates but suffer disproportionately from false positives versus false negatives.
Key Takeaway: Always disaggregate confusion matrix metrics by demographic group—equal accuracy doesn't mean equal treatment when error types differ systematically across populations.