Sequential Testing and Early Stopping
What You'll Learn
You'll learn how to make informed decisions about peeking at A/B test results before the planned end date using principled statistical methods that control error rates.
The Temptation to Peek
In traditional hypothesis testing, you fix your sample size upfront and evaluate once. But in production ML systems, stakeholders often want to check results early—maybe your new model is clearly winning or losing after a few days. The problem? Each time you peek and consider stopping, you increase the chance of a false positive (Type I error), even if you use the same 0.05 threshold.
Think of it like this: If you flip a coin 100 times but check after every 10 flips to see if it looks unfair, you'll eventually see a "significant" deviation just by chance, even with a fair coin.
Sequential Probability Ratio Test (SPRT)
SPRT is a framework that lets you continuously monitor your test and stop as soon as sufficient evidence accumulates for or against your hypothesis. It compares the likelihood ratio of two hypotheses at each observation and stops when this ratio crosses predefined thresholds.
The beauty: SPRT maintains your desired error rates (alpha and beta) while typically requiring fewer samples than fixed-horizon tests.
Alpha Spending Functions
For more flexible sequential testing, alpha spending allocates your Type I error budget (usually 0.05) across multiple planned looks at the data. Common approaches:
- Pocock boundary: Spend alpha equally across all planned looks
- O'Brien-Fleming: Spend conservatively early, more liberally later
- Custom spending functions: Tailor to your business needs
Each method adjusts the p-value threshold at each peek to ensure the overall false positive rate stays at your target level.
When to Use Sequential Testing
Best for scenarios where early detection of large effects is valuable (harm detection, rapid iteration) and you can commit to stopping rules upfront—not peeking arbitrarily whenever convenient.
Key Takeaway: Sequential testing and alpha spending let you peek at A/B test results early without inflating false positive rates, but require pre-committing to when and how you'll look at the data.