Course contentsShow
← Machine Learning and Deep Learning
Lesson 3322 of 3,53872. Bias and FairnessFree lesson

Error Analysis by Subgroup

Investigating false positive and false negative rates across groups to identify systematic prediction failures.

Error Analysis by Subgroup

What you'll learn: How to systematically investigate false positive and false negative rates across demographic groups to uncover where your model fails specific subpopulations.

The Core Idea

After computing overall fairness metrics, you need to dig deeper. Error analysis by subgroup means examining which types of mistakes your model makes for which groups. A model might have similar overall accuracy across groups but fail in completely different ways—misclassifying Group A as false positives while misclassifying Group B as false negatives.

Think of it like a medical diagnostic test: if the test frequently misses cancer in younger patients (false negatives) but over-diagnoses older patients (false positives), the overall accuracy might look acceptable, but the test is systematically failing both groups in harmful yet different ways.

Why This Matters

Fairness isn't just about equal error rates—it's about equal error types. A loan approval model that denies qualified applicants from Group A (false negatives) causes different harm than one that approves unqualified applicants from Group B (false positives). Both groups face unfairness, but the nature of that unfairness differs.

The Analysis Process

  1. Segment predictions by protected attributes (race, gender, age, etc.)
  2. Compute confusion matrices for each subgroup separately
  3. Compare FPR and FNR across groups: are certain groups experiencing systematically higher rates of specific error types?
  4. Investigate intersections: examine combinations like "young women" or "older men from rural areas"
  5. Identify patterns: which features or input characteristics correlate with these subgroup-specific failures?

This granular view reveals systematic prediction failures—patterns where your model consistently misjudges certain groups in particular ways, often reflecting training data biases or problematic learned associations.

Key Takeaway: Error analysis by subgroup exposes not just whether fairness disparities exist, but how your model fails different groups, enabling targeted mitigation strategies for specific error patterns.