Course contentsShow
← Machine Learning and Deep Learning
Lesson 92 of 3,5381. Mathematical Foundations for Machine LearningFree lesson

Multiple Testing Correction

Adjusting significance thresholds when performing multiple hypothesis tests to control error rates.

What You'll Learn

You'll learn why performing many hypothesis tests at once inflates your chance of false discoveries, and how to adjust your significance thresholds to keep errors under control.

The Problem: Testing Many Things at Once

Imagine you're testing 100 different features to see if they predict an outcome, using a significance level of α = 0.05 for each test. Even if none of the features truly matter, you'd expect about 5 false positives just by random chance (5% of 100). The more tests you run, the more likely you are to incorrectly reject a true null hypothesis—a Type I error.

This is the multiple testing problem: your overall error rate balloons when you perform many tests simultaneously.

The Solution: Adjust Your Threshold

Multiple testing correction methods lower the bar for what counts as "significant" to maintain control over the family-wise error rate (FWER) or false discovery rate (FDR).

Common Approaches

Bonferroni Correction: Divide your significance level by the number of tests. If you're doing 100 tests at α = 0.05, use α = 0.05/100 = 0.0005 for each individual test. Simple but conservative—it reduces power to detect true effects.

Holm's Method: A less conservative step-down procedure that adjusts thresholds sequentially based on ranked p-values.

Benjamini-Hochberg (FDR Control): Controls the expected proportion of false discoveries among your rejections, rather than the probability of any false discovery. More powerful when you expect some true positives.

Why It Matters in ML

In feature selection, model comparison, or hyperparameter tuning, you often evaluate many hypotheses. Without correction, you risk selecting spurious features or claiming improvements that are just noise.

Key Takeaway: When performing multiple hypothesis tests, adjust your significance thresholds using methods like Bonferroni or Benjamini-Hochberg to control the rate of false positives across all tests.