Course contentsShow
← Data Science
Lesson 1523 of 2,14537. A/B TestingFree lesson

Peeking at Results Early

Why continuous monitoring inflates false positive rates and how proper stopping rules preserve validity.

Peeking at Results Early

You'll learn why checking your A/B test results before the planned end time can mislead you into making the wrong decision.

The Temptation to Peek

Imagine running an A/B test with a planned sample size of 10,000 users. On day three, you've collected 2,000 observations and the treatment group shows a statistically significant lift. Should you stop the test and declare victory?

The urge is strong, but peeking inflates your false positive rate—sometimes dramatically.

Why Peeking is Dangerous

When you check results multiple times during a test, you're essentially conducting multiple hypothesis tests on the same experiment. Each peek gives randomness another chance to produce a "significant" result purely by luck.

Think of it like flipping a coin repeatedly. If you check whether you've gotten 3 heads in a row after every flip, you'll eventually see that pattern—even though each individual sequence is random. Similarly, metrics naturally fluctuate. Early in a test, small samples produce noisy estimates that often show "significant" differences that vanish with more data.

Your nominal 5% false positive rate (α = 0.05) can balloon to 20% or higher with frequent peeking, depending on how many times you check and when.

Proper Stopping Rules Preserve Validity

To maintain test integrity, you must either:

  • Wait until your pre-planned sample size is reached (fixed-horizon testing)
  • Use sequential testing methods specifically designed for interim analysis (like Group Sequential Testing or Always-Valid Inference from earlier lessons)

These methods adjust significance thresholds to account for multiple looks at the data, preserving your target false positive rate.

Key Takeaway: Checking A/B test results before reaching your planned sample size without using proper sequential methods multiplies your chances of false positives; always use pre-defined stopping rules.