Course contentsShow
← Data Science
Lesson 1505 of 2,14537. A/B TestingFree lesson

False Discovery Rate (FDR)

Understanding FDR as the expected proportion of false positives among rejected hypotheses, a less conservative alternative to FWER.

False Discovery Rate (FDR)

You'll learn a smarter way to control false positives when running many A/B tests at once.

When you run multiple A/B tests simultaneously (testing different button colors, headlines, and layouts), you've already learned that the multiple testing problem inflates your chance of false positives. The Bonferroni and Holm-Bonferroni corrections control the Family-Wise Error Rate (FWER)—the probability of making any false discovery—but they're extremely conservative, especially with many tests.

A More Practical Alternative

False Discovery Rate (FDR) takes a different, more permissive approach. Instead of controlling the probability of any false positive, FDR controls the expected proportion of false discoveries among all the hypotheses you reject.

Think of it this way: Suppose you run 100 A/B tests and declare 20 of them "winners." FWER asks: "What's the chance that at least one of those 20 is wrong?" FDR asks: "On average, how many of those 20 winners are false positives?"

If you set FDR = 0.05, you're accepting that about 5% of your declared "winners" might be false alarms. That's 1 out of 20 in this example. You're trading some false positives for more discoveries.

Why Use FDR?

FDR is less conservative than FWER-controlling methods. It's ideal when:

  • You're running many tests and expect several true effects
  • Missing real discoveries is costly
  • You can tolerate a small proportion of false positives among your findings

For example, if you're exploring 50 product variations and willing to accept that 1-2 "winning" variants might be flukes, FDR is more powerful than Bonferroni while still maintaining statistical rigor.

Key Takeaway: FDR controls the expected proportion of false positives among rejected hypotheses, making it a practical, less conservative alternative to FWER when discovering multiple true effects matters more than avoiding any false positive.