Sequential Testing Overview
You'll learn how sequential testing methods enable safe, continuous monitoring of A/B tests without inflating your error rates—solving the "peeking problem" inherent in fixed-horizon tests.
The Core Problem
In the previous lesson, you learned that fixed-horizon tests require you to decide your sample size upfront and wait until the end to analyze results. But what if you want to check progress along the way? What if you could stop early when results are clear?
Traditional frequentist tests break down when you "peek" at results multiple times. Each peek is essentially a new hypothesis test, compounding your Type I error rate (false positives). If you check your results 10 times during an experiment designed for α = 0.05, your actual false positive rate could climb to 20% or higher!
What Sequential Testing Offers
Sequential testing (also called sequential analysis or continuous monitoring) provides statistical methods that account for continuous or repeated looks at accumulating data. These methods adjust the decision thresholds dynamically to maintain valid error control throughout the experiment's duration.
Think of it like adjusting your security alarm sensitivity: if you're checking your house camera every hour instead of once per day, you need stricter rules about when to sound the alarm to avoid constant false alerts.
Sequential methods allow you to:
- Stop early when evidence is strong (saving time and resources)
- Peek frequently without invalidating your test
- Maintain valid α and power guarantees
- Make decisions faster on average
Common Sequential Approaches
While specific methods will come later, sequential testing encompasses techniques like Sequential Probability Ratio Tests (SPRT), group sequential designs, and alpha-spending functions.
Key Takeaway: Sequential testing methods let you monitor A/B tests continuously while controlling error rates, solving the "peeking problem" that invalidates traditional fixed-horizon tests when you look at results early.