Course contentsShow
← Data Science
Lesson 1496 of 2,14537. A/B TestingFree lesson

The Four Parameters of Sample Size Calculation

Explore the relationship between significance level, power, effect size, and required sample size.

The Four Parameters of Sample Size Calculation

What you'll learn: How significance level, statistical power, effect size, and sample size are mathematically interconnected in A/B testing.

The Four-Way Relationship

Sample size calculation isn't arbitrary—it's determined by balancing four interdependent parameters. Change one, and the others must adjust. Think of it like a seesaw with four seats: when one moves, the whole system rebalances.

The Four Parameters

  1. Significance Level (α): Your tolerance for false positives—typically 0.05. This is your Type I error rate: claiming a difference exists when it doesn't.

  2. Statistical Power (1 - β): Your ability to detect real effects—typically 0.80. Power of 80% means you'll catch a true effect 80% of the time. β is the Type II error rate.

  3. Effect Size (δ): The minimum detectable difference you care about—determined by your MDE (Minimum Detectable Effect). Smaller effects are harder to spot.

  4. Sample Size (n): The number of observations per group needed to satisfy the other three parameters.

How They Trade Off

The key insight: You can only set three parameters freely—the fourth is determined by the other three.

  • Want to detect a smaller effect? You need a larger sample (holding α and power constant)
  • Want higher power? You need a larger sample (holding α and effect size constant)
  • Want a lower significance level (stricter)? You need a larger sample (holding power and effect size constant)

This is why sample size calculators ask for α, power, and effect size as inputs—they're solving for n.

Key Takeaway: Sample size, significance level, power, and effect size form an interconnected system where fixing any three mathematically determines the fourth—understanding these trade-offs is essential for designing properly powered experiments.