Course contentsShow
← Data Science
Lesson 1493 of 2,14537. A/B TestingFree lesson

Why Sample Size Matters in A/B Tests

Understand how sample size affects statistical power, the risk of false results, and test duration.

Why Sample Size Matters in A/B Tests

You'll learn how sample size directly influences your test's ability to detect real effects, avoid misleading conclusions, and determine how long you need to run your experiment.

The Core Problem

Sample size is the foundation of statistical reliability in A/B testing. With too few observations, you risk three critical failures:

  1. Low statistical power – You might miss real effects (false negatives). Power is your probability of detecting an effect when it truly exists. Small samples mean low power, so even if your treatment genuinely improves conversion by 5%, you might conclude "no difference" simply because you didn't have enough data.

  2. High risk of false positives – Small samples produce noisy estimates. You might see a 10% lift purely by chance, declare victory, and roll out a change that actually does nothing (or worse, harms performance).

  3. Impractical test duration – If your required sample size is large but your daily traffic is small, you'll need to run the test for weeks or months. This extends decision timelines and increases exposure to external validity threats (seasonality, competitor actions, etc.).

The Analogy

Think of sample size like polling accuracy. Asking 10 people their opinion gives wildly variable results—one random group might be 80% positive, another 20%. But survey 10,000 people, and your estimate stabilizes around the true population value. A/B tests work identically: more observations = more stable, trustworthy conclusions.

Why It Connects to Everything

You already learned about Minimum Detectable Effect (MDE) and statistical power—sample size is the lever that controls both. Larger samples let you detect smaller effects with higher confidence, while smaller samples force you to only catch large, obvious changes.

Key Takeaway: Sample size determines whether your A/B test can reliably detect the effects you care about, how long the test must run, and whether your conclusions are trustworthy—always calculate required sample size before launching.