Sample Size Calculation
What you'll learn: How to determine the minimum number of observations needed in an A/B test to reliably detect a meaningful difference between model variants.
The Core Problem
After designing an A/B test, you face a critical question: How many users (samples) do I need to observe before I can trust my results? Too few, and you might miss real improvements or falsely declare success. Too many wastes time and resources.
Sample size calculation answers this by balancing three factors:
The Three Key Parameters
1. Effect Size (δ)
The minimum difference you care about detecting. If Model B improves click-through rate from 10% to 11%, your effect size is 1 percentage point. Smaller effects require more samples to detect reliably.
2. Significance Level (α)
Your tolerance for false positives (Type I error). Standard practice uses α = 0.05, meaning you accept a 5% chance of declaring a difference when none exists.
3. Statistical Power (1 - β)
The probability you'll detect a real effect when it exists. Typical power is 0.80 (80%), meaning you accept a 20% chance of missing a real improvement (Type II error).
The Calculation
For comparing two proportions (common in ML A/B tests):
n ≈ 2 × [(Z_α/2 + Z_β)² × p(1-p)] / δ²
Where p is the baseline conversion rate, and Z values come from the standard normal distribution.
Intuition: Detecting a 0.1% lift in a 50% baseline requires far more samples than detecting a 10% lift in a 1% baseline. The signal-to-noise ratio determines your needs.
Key Takeaway: Sample size calculation prevents underpowered experiments that waste resources and overpowered ones that delay deployment—calculate it before running your A/B test, not after.