Course contentsShow
← Data Science
Lesson 1512 of 2,14537. A/B TestingFree lesson

Group Sequential Testing

Testing at predetermined looks with adjusted significance levels using alpha spending functions to control overall error.

Group Sequential Testing

What you'll learn: How to check experiment results at multiple planned intervals without inflating your false positive rate.

The Core Idea

Imagine you're running a month-long A/B test, but you don't want to wait the full 30 days before peeking at results. Group Sequential Testing lets you look at your data multiple times at predetermined checkpoints (e.g., after 1 week, 2 weeks, 3 weeks, and final results) while controlling the overall Type I error rate.

The challenge? Each time you peek and run a hypothesis test, you risk a false positive. Look five times with α = 0.05 each time, and your actual error rate could approach 25%—not 5%!

How It Works

Group Sequential Testing uses alpha spending functions to divide your significance budget across multiple looks. Think of your α = 0.05 as a limited resource you must allocate carefully across checkpoints.

Common spending strategies:

  • O'Brien-Fleming: Spends very little alpha early (conservative early looks), saving most for the final analysis
  • Pocock: Distributes alpha more evenly across all looks
  • Custom functions: Tailor spending to your business needs

At each predetermined checkpoint, you compare your test statistic against an adjusted critical value. If you cross the boundary, you can stop early for efficacy; otherwise, continue to the next look.

Why It Matters

This approach gives you the flexibility to monitor experiments and potentially stop early (saving time and resources) while maintaining rigorous statistical control. The key is pre-specifying when you'll look and committing to the spending function before seeing any data.

Key Takeaway: Group Sequential Testing allows multiple interim analyses at planned intervals using alpha spending functions to distribute your significance level, keeping overall Type I error controlled at your chosen α.