Course contentsShow
← Data Science
Lesson 337 of 2,1459. Hypothesis Testing FoundationsFree lesson

Error Rates in Practice: Multiple Testing

Understand how Type I error rates inflate when conducting multiple hypothesis tests without correction.

Error Rates in Practice: Multiple Testing

What you'll learn: How running many hypothesis tests at once inflates your false positive rate far beyond your chosen α level.

The Multiple Testing Problem

Imagine you set α = 0.05, meaning you're willing to accept a 5% chance of a false positive (Type I error) on one test. That sounds reasonable—but what happens when you run many tests?

If you conduct 20 independent hypothesis tests, each at α = 0.05, the probability of getting at least one false positive is much higher than 5%. In fact, it's approximately:

1 - (0.95)^20 ≈ 0.64 or 64%

You're almost guaranteed to find something "significant" by chance alone!

Why This Matters in Practice

Multiple testing scenarios are everywhere in data science:

  • Testing 50 different features to see which correlate with an outcome
  • Running A/B tests on multiple variations simultaneously
  • Analyzing survey data with dozens of questions
  • Gene expression studies examining thousands of genes

Each test carries its own α risk. When you don't adjust for multiple comparisons, your family-wise error rate (the probability of making any Type I error across all tests) balloons.

The Core Issue

Your nominal α = 0.05 only applies to a single test. Running multiple tests without correction turns your study into a "fishing expedition" where spurious findings become likely. This is why correction methods (like Bonferroni or false discovery rate control) exist—but understanding the problem comes first.

Key Takeaway: When conducting multiple hypothesis tests, your effective Type I error rate compounds across tests, making false discoveries far more likely than your chosen α suggests—demanding correction strategies to maintain valid inference.