Always-Valid Inference and Confidence Sequences
What you'll learn: How modern sequential methods let you peek at your A/B test anytime without inflating error rates.
The Problem with Peeking
Traditional hypothesis tests assume you'll look at your data exactly once, at a pre-planned sample size. But in practice, you often want to check results early — and every "peek" inflates your false positive rate. Group Sequential Testing (lesson 1512) controlled this with pre-specified looks, but what if you want ultimate flexibility?
Always-Valid Inference: Stop Whenever You Want
Always-valid inference provides p-values and confidence intervals that remain statistically valid no matter when you stop — whether you check once, continuously, or at random times you didn't plan ahead. This is the ultimate solution to optional stopping.
The key innovation is confidence sequences: intervals that evolve over time and contain the true parameter with at least 1-α probability simultaneously at all sample sizes.
How It Works
Instead of a single confidence interval at a fixed n, you get a sequence of intervals [L₁, U₁], [L₂, U₂], ..., one for each data point. These are constructed so that:
- P(true value ∈ [Lₜ, Uₜ] for all t) ≥ 1-α
Think of it like a safety tunnel: the true value stays inside the entire path, not just at one checkpoint.
Example intuition: In an A/B test for conversion rates, your confidence sequence might start wide and narrow over time. You can stop whenever the sequence excludes zero (indicating significance) — no penalty for checking early or often.
Key Takeaway: Always-valid inference and confidence sequences enable flexible stopping in experiments without sacrificing statistical validity, making them ideal for real-world A/B testing where you can't always plan every analysis checkpoint in advance.