Course contentsShow
← Machine Learning and Deep Learning
Lesson 3074 of 3,53867. A/B Testing and Deployment StrategiesFree lesson

Multiple Testing Problem and Corrections

Understand the risks of multiple comparisons and apply Bonferroni, FDR, and other correction methods.

Multiple Testing Problem and Corrections

What you'll learn: How running multiple A/B tests simultaneously inflates your false positive rate and how to mathematically correct for it.

The Problem: More Tests, More False Alarms

When you run a single A/B test with α = 0.05, you accept a 5% chance of falsely detecting an effect. But what happens when you test 20 different metrics simultaneously? By pure chance, you'd expect one "significant" result even if nothing real changed—like flipping 20 coins and celebrating when one lands on heads 10 times.

This is the multiple testing problem: each additional test increases your family-wise error rate (FWER)—the probability of making at least one false discovery across all tests.

Correction Methods

Bonferroni Correction

The most conservative approach: divide your significance threshold by the number of tests.

  • Testing 10 metrics? Use α = 0.05/10 = 0.005 for each
  • Pro: Simple, controls FWER strictly
  • Con: Very conservative; reduces statistical power

False Discovery Rate (FDR)

Controls the proportion of false positives among your significant results, not the probability of any false positive. The Benjamini-Hochberg procedure ranks p-values and applies adaptive thresholds.

  • Pro: More power than Bonferroni
  • Con: Allows some false positives

Holm-Bonferroni

A step-down method that's less conservative than standard Bonferroni while still controlling FWER.

Practical Application

When deploying a new model and monitoring 50 metrics, you must correct for multiple comparisons or you'll trigger false alarms constantly. Choose Bonferroni for critical safety metrics; use FDR when exploring many features for retraining signals.

Key Takeaway: Multiple testing inflates false positive rates; apply Bonferroni correction for strict error control or FDR methods when you need more statistical power across many comparisons.