Early Stopping in Bayesian Tests
What you'll learn: How to stop A/B tests early in a Bayesian framework without worrying about inflating error rates.
The Freedom to Stop Early
One of Bayesian testing's biggest advantages is that you can check your results as often as you like and stop when confident—without the "peeking problem" that plagues frequentist tests. In Bayesian inference, the posterior probability represents your current belief given all data so far, and it doesn't change just because you looked multiple times.
How Early Stopping Works
You define a stopping rule based on posterior probabilities before the test begins. Common approaches include:
- Probability threshold: Stop when P(B better than A | data) > 0.95
- Expected loss threshold: Stop when the expected loss of choosing variant B is below $X
- Combination rule: Require both high probability (e.g., 95%) and low expected loss
The key insight: these rules are built into your decision-making framework from the start. You're not "testing until significance"—you're monitoring until you've gathered enough evidence for a confident business decision.
Why This Doesn't Inflate Error Rates
Unlike frequentist tests where each peek increases false positive risk, Bayesian posteriors already account for uncertainty in the data. When you say "there's a 95% probability that B is better," that statement remains valid regardless of how many times you checked. Your beliefs update continuously as data arrives; stopping is simply choosing when you have enough certainty to act.
Key Takeaway: Bayesian A/B tests allow flexible early stopping by defining posterior probability or expected loss thresholds upfront—you can monitor continuously and stop when you reach your decision criterion without inflating error rates, because the posterior probability directly quantifies your uncertainty given all observed data.