Course contentsShow
← Data Science
Lesson 1509 of 2,14537. A/B TestingFree lesson

The Problem with Fixed-Horizon Tests

Why stopping an A/B test early or peeking at results can inflate false positive rates beyond the nominal significance level.

The Problem with Fixed-Horizon Tests

What you'll learn: Why stopping an A/B test early or peeking at results can inflate false positive rates beyond the nominal significance level.

The Dangerous Temptation to Peek

When you design an A/B test with a pre-calculated sample size (the "fixed horizon"), you're setting up a statistical framework that assumes you'll collect all the data before making a decision. But what happens when you check results midway through?

Here's the problem: every time you peek at your test results and perform a significance test, you're essentially giving yourself another chance to find a "significant" result by random chance alone. If you set α = 0.05, you're accepting a 5% false positive rate per test. But if you peek 5 times during the experiment, you're not actually maintaining that 5% error rate anymore—you're inflating it, potentially to 15% or higher.

Why This Happens

Think of it like flipping a coin until you get heads. If you flip once, you have a 50% chance. But if you keep flipping and can stop whenever you want, you'll eventually get heads with near certainty. Similarly, randomly fluctuating A/B test results will occasionally cross the significance threshold just by chance—and if you're watching continuously, you might stop exactly when that happens.

This violates the fixed-horizon assumption: that you predetermined the sample size and committed to collecting all data before deciding.

The Real-World Consequence

Your carefully calculated 5% significance level becomes meaningless. You might declare winners that aren't real, ship changes that don't actually improve metrics, and erode trust in your experimentation program.

Key Takeaway: Peeking at A/B test results before reaching your predetermined sample size inflates your false positive rate beyond your nominal α level, compromising the statistical validity of your test.