Course contentsShow
← Data Science
Lesson 1514 of 2,14537. A/B TestingFree lesson

Practical Implementation of Sequential Tests

How to calculate adjusted boundaries, determine monitoring frequency, and implement sequential stopping rules in practice.

Practical Implementation of Sequential Tests

What You'll Learn

You'll learn how to operationalize sequential tests by calculating adjusted decision boundaries, setting monitoring schedules, and implementing stopping rules in real A/B testing platforms.

From Theory to Practice

Sequential testing methods like SPRT, group sequential designs, and always-valid inference give you the framework, but implementing them requires concrete operational decisions. You need to translate mathematical boundaries into code and workflow.

Calculating Adjusted Boundaries

Rather than using the standard significance threshold (α = 0.05), sequential tests require adjusted boundaries that account for repeated looks at the data. For group sequential tests, you might use an α-spending function (like O'Brien-Fleming) that allocates very little Type I error to early looks and more to later ones. This means your early decision threshold might be p < 0.001, gradually relaxing toward 0.05 at the final analysis.

For always-valid confidence sequences, you compute time-varying confidence intervals that remain valid no matter when you peek.

Determining Monitoring Frequency

How often should you check? More frequent checks require stricter boundaries. In group sequential designs, you pre-specify exactly 3–5 interim analyses (e.g., at 25%, 50%, 75%, 100% of planned sample). For continuous monitoring (anytime-valid tests), you can check daily but must use appropriate always-valid boundaries that account for unlimited peeks.

Implementing Stopping Rules

Your code should automatically:

  • Calculate the current test statistic and adjusted boundary
  • Stop for efficacy if you cross the upper boundary (strong positive effect)
  • Stop for futility if the effect is clearly too small to matter
  • Continue collecting data if inconclusive
  • Enforce a maximum sample size or duration

Key Takeaway: Implementing sequential tests means choosing your monitoring schedule upfront, computing time-adjusted significance boundaries using appropriate corrections, and automating stopping decisions based on pre-registered rules—not eyeballing dashboards and making subjective calls.