Course contentsShow
← Data Science
Lesson 1513 of 2,14537. A/B TestingFree lesson

Always-Valid Inference and Confidence Sequences

Modern methods that provide valid p-values and confidence intervals at any stopping time without pre-specified looks.

Always-Valid Inference and Confidence Sequences

What you'll learn: How modern sequential methods let you peek at your A/B test anytime without inflating error rates.

The Problem with Peeking

Traditional hypothesis tests assume you'll look at your data exactly once, at a pre-planned sample size. But in practice, you often want to check results early — and every "peek" inflates your false positive rate. Group Sequential Testing (lesson 1512) controlled this with pre-specified looks, but what if you want ultimate flexibility?

Always-Valid Inference: Stop Whenever You Want

Always-valid inference provides p-values and confidence intervals that remain statistically valid no matter when you stop — whether you check once, continuously, or at random times you didn't plan ahead. This is the ultimate solution to optional stopping.

The key innovation is confidence sequences: intervals that evolve over time and contain the true parameter with at least 1-α probability simultaneously at all sample sizes.

How It Works

Instead of a single confidence interval at a fixed n, you get a sequence of intervals [L₁, U₁], [L₂, U₂], ..., one for each data point. These are constructed so that:

  • P(true value ∈ [Lₜ, Uₜ] for all t) ≥ 1-α

Think of it like a safety tunnel: the true value stays inside the entire path, not just at one checkpoint.

Example intuition: In an A/B test for conversion rates, your confidence sequence might start wide and narrow over time. You can stop whenever the sequence excludes zero (indicating significance) — no penalty for checking early or often.

Key Takeaway: Always-valid inference and confidence sequences enable flexible stopping in experiments without sacrificing statistical validity, making them ideal for real-world A/B testing where you can't always plan every analysis checkpoint in advance.