Course contentsShow
← Data Science
Lesson 1501 of 2,14537. A/B TestingFree lesson

The Multiple Testing Problem

Understanding how conducting multiple hypothesis tests inflates the overall Type I error rate beyond the nominal significance level.

The Multiple Testing Problem

What You'll Learn

You'll learn how running many hypothesis tests at once inflates your chance of false positives beyond your chosen significance level.

The Core Problem

Imagine you're running an A/B test at α = 0.05 (5% significance level). This means you accept a 5% chance of a false positive—claiming a difference exists when it doesn't (Type I error).

Now suppose you test not just one metric, but 20 different metrics: click-through rate, time on page, bounce rate, revenue per user, etc. Each individual test has a 5% false positive rate, but what's the probability that at least one of those 20 tests produces a false positive purely by chance?

The math is sobering: with 20 independent tests at α = 0.05, the probability of at least one false positive is approximately 1 - (0.95)^20 ≈ 0.64, or 64%!

Why This Happens

Each test is like a coin flip with a small chance of landing on "significant by luck." The more tests you run, the more opportunities you create for random noise to look like a real effect. Your family-wise error rate (the probability of making any Type I error across all tests) balloons far beyond your nominal 5%.

Real-World Example

You run an A/B test and check 10 engagement metrics. One shows p = 0.03—"significant!" But did you discover something real, or did you just get unlucky on 1 out of 10 tries? Without correction, you can't tell.

Key Takeaway: Running multiple hypothesis tests without adjustment dramatically increases your overall Type I error rate, turning your 5% significance level into a much higher chance of false discoveries—a critical issue when testing multiple metrics in A/B experiments.