The Four Parameters of Sample Size Calculation
What you'll learn: How significance level, statistical power, effect size, and sample size are mathematically interconnected in A/B testing.
The Four-Way Relationship
Sample size calculation isn't arbitrary—it's determined by balancing four interdependent parameters. Change one, and the others must adjust. Think of it like a seesaw with four seats: when one moves, the whole system rebalances.
The Four Parameters
-
Significance Level (α): Your tolerance for false positives—typically 0.05. This is your Type I error rate: claiming a difference exists when it doesn't.
-
Statistical Power (1 - β): Your ability to detect real effects—typically 0.80. Power of 80% means you'll catch a true effect 80% of the time. β is the Type II error rate.
-
Effect Size (δ): The minimum detectable difference you care about—determined by your MDE (Minimum Detectable Effect). Smaller effects are harder to spot.
-
Sample Size (n): The number of observations per group needed to satisfy the other three parameters.
How They Trade Off
The key insight: You can only set three parameters freely—the fourth is determined by the other three.
- Want to detect a smaller effect? You need a larger sample (holding α and power constant)
- Want higher power? You need a larger sample (holding α and effect size constant)
- Want a lower significance level (stricter)? You need a larger sample (holding power and effect size constant)
This is why sample size calculators ask for α, power, and effect size as inputs—they're solving for n.
Key Takeaway: Sample size, significance level, power, and effect size form an interconnected system where fixing any three mathematically determines the fourth—understanding these trade-offs is essential for designing properly powered experiments.