Statistical Significance and Test Duration
You'll learn how to determine if your A/B test results are real improvements or just random noise, and how long to run tests for reliable conclusions.
The Core Challenge
After deploying two model variants in an A/B test, you need to answer: "Is Model B actually better, or did I just get lucky?" Running tests too short risks false positives—declaring a winner when none exists. Running them too long wastes resources and delays rollout.
Statistical significance tells you the probability that observed differences aren't due to chance. The industry standard is p < 0.05, meaning less than 5% chance your results are random.
Sample Size Calculation
Before launching, calculate the minimum sample size needed to detect a meaningful difference. This depends on:
- Baseline metric (e.g., current 80% success rate)
- Minimum detectable effect (e.g., want to catch 2% improvement)
- Statistical power (typically 80%—ability to detect real differences)
- Significance level (typically 0.05)
Think of it like weighing packages: detecting a 10-gram difference needs a more sensitive scale than detecting 1 kilogram. Smaller differences require larger samples.
Test Duration
Run tests until both conditions are met:
- You've reached minimum sample size
- You've captured typical usage patterns (weekdays, weekends, time zones)
Don't stop early just because results look good—this introduces peeking bias. A model might appear better on Monday but worse averaged over a full week.
For low-traffic systems, duration matters more than raw sample count. Even with enough requests, you need temporal coverage.
Key Takeaway: Calculate required samples upfront based on the smallest improvement worth detecting, then run tests through complete usage cycles before declaring statistical significance—rushing leads to false conclusions that waste deployment effort.