Course contentsShow
← Machine Learning and Deep Learning
Lesson 3300 of 3,53872. Bias and FairnessFree lesson

Confusion Matrix Disparities

Analyzing how error rates (FPR, FNR, precision, recall) differ across demographic groups.

Confusion Matrix Disparities

What you'll learn: How different demographic groups can experience dramatically different types of prediction errors, even when overall accuracy looks fair.

The Hidden Inequality in Error Types

When evaluating fairness, overall accuracy can mask critical disparities. A model might achieve 85% accuracy for both Group A and Group B, yet treat them very differently in how it makes mistakes.

The confusion matrix breaks predictions into four categories: True Positives (TP), False Positives (FP), True Negatives (TN), and False Negatives (FN). Confusion matrix disparities occur when error rates derived from these cells differ significantly across demographic groups.

Why This Matters

Consider a loan approval model:

  • Group A might face a high False Positive Rate (wrongly denied loans they could repay)
  • Group B might face a high False Negative Rate (wrongly approved for loans they'll default on)

Both harm people differently. High FPR denies opportunities; high FNR leads to debt traps.

Key Metrics to Compare

When analyzing confusion matrix disparities, compute these rates separately for each group:

  • False Positive Rate (FPR): FP / (FP + TN) — how often negatives are misclassified
  • False Negative Rate (FNR): FN / (FN + TP) — how often positives are missed
  • Precision: TP / (TP + FP) — accuracy of positive predictions
  • Recall: TP / (TP + FN) — coverage of actual positives

Large gaps in these metrics between groups signal that your model distributes harm unevenly, even if aggregate performance seems balanced.

The Insight

A "fair" model by one metric (say, Demographic Parity) can still exhibit severe confusion matrix disparities. A group might receive equal positive prediction rates but suffer disproportionately from false positives versus false negatives.

Key Takeaway: Always disaggregate confusion matrix metrics by demographic group—equal accuracy doesn't mean equal treatment when error types differ systematically across populations.