Understanding When Test Results Are Reliable and How Long Tests Should Run
You'll learn how to determine whether your A/B test results are trustworthy and when you have enough data to make confident decisions.
The Core Problem
When running A/B tests with feature flags, you might see one variant performing better after just a few dozen users. But is that difference real, or just random luck? Statistical significance tells you whether your test results are meaningful enough to trust, while sample size determines how long you need to run the test.
Statistical Significance Explained
Think of flipping a coin: if you flip twice and get two heads, you can't conclude the coin is biased. But after 1,000 flips showing 750 heads, you'd be confident something's wrong. Statistical significance works the same way.
In A/B testing, we typically aim for 95% confidence (p-value < 0.05), meaning there's only a 5% chance the observed difference happened by random chance. This threshold prevents you from declaring "winner!" when you've just gotten lucky with early data.
Sample Size Requirements
Your required sample size depends on:
- Expected effect size: Smaller differences (like 2% vs 2.1% conversion) need far more users than dramatic ones (2% vs 4%)
- Baseline conversion rate: Tests with very low conversion rates need longer runs
- Desired confidence level: Higher confidence requires more data
- Traffic distribution: 50/50 splits reach significance faster than 90/10 splits
Most A/B testing tools calculate this automatically, but plan for at least 1-2 weeks minimum and several hundred conversions per variant for reliable results, even if significance is reached earlier (to account for day-of-week and time-based variations).
Key Takeaway: Never stop tests early just because you see significance—wait for both statistical significance AND your planned sample size to avoid false positives and ensure decisions based on real user behavior patterns.