Course contentsShow
← Data Science
Lesson 1515 of 2,14537. A/B TestingFree lesson

Trade-offs: Sample Size, Speed, and Complexity

Understanding when sequential testing reduces sample size versus fixed-horizon tests and the operational complexity involved.

Trade-offs: Sample Size, Speed, and Complexity

You'll learn when sequential testing saves you samples versus fixed-horizon approaches, and what operational complexity you accept in return.

The Central Trade-off

Sequential testing methods like SPRT, group sequential designs, and always-valid inference promise one major benefit: reaching conclusions faster with potentially fewer samples than fixed-horizon tests. But this speed comes at a cost: increased operational and analytical complexity.

Sample Size and Speed Advantages

Sequential tests shine when there's a strong treatment effect. You can stop early and declare a winner without waiting for the full planned sample. In practice, this can mean:

  • 20-50% fewer samples when effects are large
  • Days or weeks saved in fast-moving business environments
  • Reduced opportunity cost of running inferior variants

When effects are small or null, sequential tests may run as long (or slightly longer) than fixed-horizon tests, but you haven't lost much.

Complexity Costs

The price you pay:

  • Monitoring infrastructure: You need systems to check stopping conditions regularly (daily, hourly, or continuously)
  • Statistical sophistication: Teams must understand alpha spending functions, confidence sequences, or group boundaries—not just basic t-tests
  • Cultural buy-in: Stakeholders may question "peeking" at results, requiring education
  • Edge cases: Handling tests that never stop, or stopping rules that trigger unexpectedly

When to Choose Which

Use fixed-horizon tests when you have limited statistical resources, simple infrastructure, or tests that naturally end (seasonal campaigns). Use sequential tests when speed matters, you run many experiments, and you have the infrastructure to monitor them properly.

Key Takeaway: Sequential testing can significantly reduce sample size when effects are strong, but requires investing in monitoring systems and statistical expertise—choose based on your organization's maturity and needs.