Benjamini-Hochberg Procedure
What you'll learn: How to control the False Discovery Rate using a step-up procedure that lets you find more significant results while keeping false positives in check.
The Core Idea
When you're running multiple A/B tests (say, 20 different variants), the Bonferroni and Holm-Bonferroni methods protect you from any false positives—but they're extremely conservative. You might miss real effects because you're being too strict.
The Benjamini-Hochberg (BH) procedure takes a different approach. Instead of controlling the Family-Wise Error Rate (FWER), it controls the False Discovery Rate (FDR)—the proportion of your "significant" findings that are actually false positives. If you can tolerate 5% of your discoveries being wrong, BH lets you discover much more.
How the Step-Up Procedure Works
Here's the recipe:
- Order your p-values from smallest to largest: p₁ ≤ p₂ ≤ ... ≤ pₘ
- For each p-value, calculate its threshold: (i/m) × α, where i is its rank, m is the total number of tests, and α is your target FDR level (e.g., 0.05)
- Find the largest i where pᵢ ≤ (i/m) × α
- Reject all hypotheses from 1 through i
Analogy: Think of it like a staircase of acceptance thresholds—each test gets a slightly more generous bar. You climb up until you find the last test that passes, then accept everything below it.
Why It Works
The step-up design ensures that even if some of your "discoveries" are false, the overall proportion stays below your FDR threshold. You're trading strict certainty for more discoveries.
Key Takeaway: The Benjamini-Hochberg procedure controls FDR using a step-up approach, allowing more true positives than FWER methods while maintaining an acceptable rate of false discoveries—ideal for exploratory A/B testing with many variants.