Course contentsShow
← Data Science
Lesson 1503 of 2,14537. A/B TestingFree lesson

Bonferroni Correction

Controlling FWER by dividing the significance level by the number of tests; understanding its conservative nature and when to use it.

Bonferroni Correction

What you'll learn: How to control the family-wise error rate by adjusting your significance threshold when running multiple A/B tests simultaneously.

The Core Problem

When you run multiple hypothesis tests at once (say, testing 5 different button colors in an A/B test), your chance of getting at least one false positive increases dramatically. If you test at α = 0.05 for each test, you're actually allowing a much higher overall error rate across all tests combined.

The Bonferroni Solution

The Bonferroni correction is beautifully simple: divide your desired significance level by the number of tests you're running.

If you want to maintain an overall α = 0.05 across 5 tests, use:

α_adjusted = 0.05 / 5 = 0.01

Now you only call a result "significant" if its p-value is below 0.01, not 0.05.

Why It's Conservative

Think of Bonferroni as the "better safe than sorry" approach. By lowering the threshold for each individual test, you're making it harder to declare any single result significant. This strongly controls FWER (you learned this in the previous lesson) but reduces statistical power—you might miss real effects because the bar is now so high.

When to Use It

  • Running a small to moderate number of tests (2–10)
  • When false positives are very costly
  • When tests are independent or you want a simple, transparent method
  • Early exploration when you need strict control

Bonferroni is less ideal when running many tests (50+), as it becomes too conservative and you lose power to detect real effects.

Key Takeaway: Bonferroni correction controls FWER by dividing α by the number of tests—simple and conservative, best for small test counts where false positives are expensive.