Course contentsShow
← Web Development
Lesson 2827 of 2,98250. Third-Party IntegrationsFree lesson

Statistical Significance and Sample Size

Understanding when test results are reliable and how long tests should run.

Understanding When Test Results Are Reliable and How Long Tests Should Run

You'll learn how to determine whether your A/B test results are trustworthy and when you have enough data to make confident decisions.

The Core Problem

When running A/B tests with feature flags, you might see one variant performing better after just a few dozen users. But is that difference real, or just random luck? Statistical significance tells you whether your test results are meaningful enough to trust, while sample size determines how long you need to run the test.

Statistical Significance Explained

Think of flipping a coin: if you flip twice and get two heads, you can't conclude the coin is biased. But after 1,000 flips showing 750 heads, you'd be confident something's wrong. Statistical significance works the same way.

In A/B testing, we typically aim for 95% confidence (p-value < 0.05), meaning there's only a 5% chance the observed difference happened by random chance. This threshold prevents you from declaring "winner!" when you've just gotten lucky with early data.

Sample Size Requirements

Your required sample size depends on:

  • Expected effect size: Smaller differences (like 2% vs 2.1% conversion) need far more users than dramatic ones (2% vs 4%)
  • Baseline conversion rate: Tests with very low conversion rates need longer runs
  • Desired confidence level: Higher confidence requires more data
  • Traffic distribution: 50/50 splits reach significance faster than 90/10 splits

Most A/B testing tools calculate this automatically, but plan for at least 1-2 weeks minimum and several hundred conversions per variant for reliable results, even if significance is reached earlier (to account for day-of-week and time-based variations).

Key Takeaway: Never stop tests early just because you see significance—wait for both statistical significance AND your planned sample size to avoid false positives and ensure decisions based on real user behavior patterns.