Trade-offs: Sample Size, Speed, and Complexity
You'll learn when sequential testing saves you samples versus fixed-horizon approaches, and what operational complexity you accept in return.
The Central Trade-off
Sequential testing methods like SPRT, group sequential designs, and always-valid inference promise one major benefit: reaching conclusions faster with potentially fewer samples than fixed-horizon tests. But this speed comes at a cost: increased operational and analytical complexity.
Sample Size and Speed Advantages
Sequential tests shine when there's a strong treatment effect. You can stop early and declare a winner without waiting for the full planned sample. In practice, this can mean:
- 20-50% fewer samples when effects are large
- Days or weeks saved in fast-moving business environments
- Reduced opportunity cost of running inferior variants
When effects are small or null, sequential tests may run as long (or slightly longer) than fixed-horizon tests, but you haven't lost much.
Complexity Costs
The price you pay:
- Monitoring infrastructure: You need systems to check stopping conditions regularly (daily, hourly, or continuously)
- Statistical sophistication: Teams must understand alpha spending functions, confidence sequences, or group boundaries—not just basic t-tests
- Cultural buy-in: Stakeholders may question "peeking" at results, requiring education
- Edge cases: Handling tests that never stop, or stopping rules that trigger unexpectedly
When to Choose Which
Use fixed-horizon tests when you have limited statistical resources, simple infrastructure, or tests that naturally end (seasonal campaigns). Use sequential tests when speed matters, you run many experiments, and you have the infrastructure to monitor them properly.
Key Takeaway: Sequential testing can significantly reduce sample size when effects are strong, but requires investing in monitoring systems and statistical expertise—choose based on your organization's maturity and needs.