Course contentsShow
← Data Science
Lesson 1506 of 2,14537. A/B TestingFree lesson

Benjamini-Hochberg Procedure

Controlling FDR through a step-up procedure that allows more discoveries while maintaining acceptable false positive rates.

Benjamini-Hochberg Procedure

What you'll learn: How to control the False Discovery Rate using a step-up procedure that lets you find more significant results while keeping false positives in check.

The Core Idea

When you're running multiple A/B tests (say, 20 different variants), the Bonferroni and Holm-Bonferroni methods protect you from any false positives—but they're extremely conservative. You might miss real effects because you're being too strict.

The Benjamini-Hochberg (BH) procedure takes a different approach. Instead of controlling the Family-Wise Error Rate (FWER), it controls the False Discovery Rate (FDR)—the proportion of your "significant" findings that are actually false positives. If you can tolerate 5% of your discoveries being wrong, BH lets you discover much more.

How the Step-Up Procedure Works

Here's the recipe:

  1. Order your p-values from smallest to largest: p₁ ≤ p₂ ≤ ... ≤ pₘ
  2. For each p-value, calculate its threshold: (i/m) × α, where i is its rank, m is the total number of tests, and α is your target FDR level (e.g., 0.05)
  3. Find the largest i where pᵢ ≤ (i/m) × α
  4. Reject all hypotheses from 1 through i

Analogy: Think of it like a staircase of acceptance thresholds—each test gets a slightly more generous bar. You climb up until you find the last test that passes, then accept everything below it.

Why It Works

The step-up design ensures that even if some of your "discoveries" are false, the overall proportion stays below your FDR threshold. You're trading strict certainty for more discoveries.

Key Takeaway: The Benjamini-Hochberg procedure controls FDR using a step-up approach, allowing more true positives than FWER methods while maintaining an acceptable rate of false discoveries—ideal for exploratory A/B testing with many variants.