Error Rates in Practice: Multiple Testing
What you'll learn: How running many hypothesis tests at once inflates your false positive rate far beyond your chosen α level.
The Multiple Testing Problem
Imagine you set α = 0.05, meaning you're willing to accept a 5% chance of a false positive (Type I error) on one test. That sounds reasonable—but what happens when you run many tests?
If you conduct 20 independent hypothesis tests, each at α = 0.05, the probability of getting at least one false positive is much higher than 5%. In fact, it's approximately:
1 - (0.95)^20 ≈ 0.64 or 64%
You're almost guaranteed to find something "significant" by chance alone!
Why This Matters in Practice
Multiple testing scenarios are everywhere in data science:
- Testing 50 different features to see which correlate with an outcome
- Running A/B tests on multiple variations simultaneously
- Analyzing survey data with dozens of questions
- Gene expression studies examining thousands of genes
Each test carries its own α risk. When you don't adjust for multiple comparisons, your family-wise error rate (the probability of making any Type I error across all tests) balloons.
The Core Issue
Your nominal α = 0.05 only applies to a single test. Running multiple tests without correction turns your study into a "fishing expedition" where spurious findings become likely. This is why correction methods (like Bonferroni or false discovery rate control) exist—but understanding the problem comes first.
Key Takeaway: When conducting multiple hypothesis tests, your effective Type I error rate compounds across tests, making false discoveries far more likely than your chosen α suggests—demanding correction strategies to maintain valid inference.