The Problem with Fixed-Horizon Tests
What you'll learn: Why stopping an A/B test early or peeking at results can inflate false positive rates beyond the nominal significance level.
The Dangerous Temptation to Peek
When you design an A/B test with a pre-calculated sample size (the "fixed horizon"), you're setting up a statistical framework that assumes you'll collect all the data before making a decision. But what happens when you check results midway through?
Here's the problem: every time you peek at your test results and perform a significance test, you're essentially giving yourself another chance to find a "significant" result by random chance alone. If you set α = 0.05, you're accepting a 5% false positive rate per test. But if you peek 5 times during the experiment, you're not actually maintaining that 5% error rate anymore—you're inflating it, potentially to 15% or higher.
Why This Happens
Think of it like flipping a coin until you get heads. If you flip once, you have a 50% chance. But if you keep flipping and can stop whenever you want, you'll eventually get heads with near certainty. Similarly, randomly fluctuating A/B test results will occasionally cross the significance threshold just by chance—and if you're watching continuously, you might stop exactly when that happens.
This violates the fixed-horizon assumption: that you predetermined the sample size and committed to collecting all data before deciding.
The Real-World Consequence
Your carefully calculated 5% significance level becomes meaningless. You might declare winners that aren't real, ship changes that don't actually improve metrics, and erode trust in your experimentation program.
Key Takeaway: Peeking at A/B test results before reaching your predetermined sample size inflates your false positive rate beyond your nominal α level, compromising the statistical validity of your test.