Course contentsShow
← Machine Learning and Deep Learning
Lesson 3323 of 3,53872. Bias and FairnessFree lesson

Statistical Significance Testing

Applying hypothesis tests to determine if observed performance differences across groups are statistically meaningful.

Statistical Significance Testing

What you'll learn: How to determine whether observed fairness disparities between groups are real signals or just random noise using hypothesis testing.

Why Statistical Significance Matters in Fairness

When you discover that Model A achieves 85% accuracy on Group X but 82% on Group Y, is that 3-point gap meaningful, or could it have happened by chance? Statistical significance testing answers this question by quantifying the probability that the observed difference occurred randomly.

This is crucial during fairness audits: you don't want to raise alarms over noise, nor dismiss genuine disparities as coincidence.

The Core Approach

Statistical significance testing applies hypothesis tests (like chi-squared tests, t-tests, or permutation tests) to performance metrics across groups:

  1. Null hypothesis (H₀): No real difference exists between groups; any observed gap is due to random sampling
  2. Alternative hypothesis (H₁): A genuine difference exists
  3. Compute a p-value: The probability of seeing a difference this large (or larger) if H₀ were true
  4. Decision rule: If p-value < threshold (typically 0.05), reject H₀ and conclude the difference is statistically significant

Practical Example

Suppose you're auditing false positive rates across two demographic groups. You observe FPR = 0.12 for Group A (n=500) and FPR = 0.15 for Group B (n=450). A two-proportion z-test can determine if this 0.03 difference is significant, accounting for sample sizes and variance.

Key Takeaway: Statistical significance testing distinguishes real fairness violations from random fluctuations by calculating whether observed performance differences across groups are unlikely to occur by chance alone—critical for making evidence-based mitigation decisions during fairness audits.