Course contentsShow
← Data Science
Lesson 1502 of 2,14537. A/B TestingFree lesson

Family-Wise Error Rate (FWER)

Defining FWER as the probability of making at least one false discovery when testing multiple hypotheses simultaneously.

Family-Wise Error Rate (FWER)

What you'll learn: You'll understand how FWER quantifies the risk of false positives when running multiple hypothesis tests in A/B testing experiments.

The Problem With Multiple Tests

When you run a single A/B test with α = 0.05, you accept a 5% chance of a false positive (Type I error). But what happens when you test multiple variations or metrics simultaneously?

Here's where things get tricky: each test carries its own error rate. The more tests you run, the more opportunities you create for at least one false discovery to slip through—even when nothing truly differs between groups.

Defining FWER

Family-Wise Error Rate (FWER) is the probability of making at least one false discovery (Type I error) across a "family" of hypothesis tests conducted simultaneously.

Think of it like this: if you flip a fair coin once, you have a 50% chance of heads. But if you flip it 10 times, the probability of getting at least one heads becomes much higher (over 99%). Similarly, testing multiple hypotheses inflates your chance of finding at least one "significant" result purely by luck.

The Math Behind It

If you conduct m independent tests, each at significance level α:

  • Probability of no false positives in one test: (1 - α)
  • Probability of no false positives across m tests: (1 - α)^m
  • FWER = 1 - (1 - α)^m

For example, with 10 independent tests at α = 0.05: FWER = 1 - (0.95)^10 ≈ 0.40 (40% chance of at least one false positive!)

Key Takeaway: FWER measures your overall risk of making at least one false discovery when testing multiple hypotheses—it's the family-level Type I error rate that grows with each additional test you conduct.