Adjusting for Baseline Conversion Rates
You'll learn how baseline conversion rates dramatically affect the sample size needed for your A/B test, and why ignoring this can lead to underpowered experiments.
Why Baseline Rates Matter
When calculating sample size for an A/B test involving proportions (like conversion rates), the baseline conversion rate — your current rate before any intervention — has a huge impact on required sample size.
Here's the intuition: detecting the same absolute change (say, +2 percentage points) is much harder when your baseline rate is near 0% or 100% than when it's around 50%.
The Variance Connection
The variance of a proportion follows the formula p(1-p). This variance is maximized when p = 0.5 and minimized at the extremes (0 or 1). Larger variance means you need fewer samples to detect the same effect size — there's more "signal" to work with.
Low baseline rate example: If your current conversion is 2%, improving to 3% (a 50% relative lift!) requires detecting a tiny absolute change. The variance is small (0.02 × 0.98 = 0.0196), so you need enormous sample sizes.
Moderate baseline rate example: If your baseline is 50%, improving to 51% has much higher variance (0.50 × 0.50 = 0.25), making the effect easier to detect with smaller samples.
Practical Planning
When planning your test, always check your baseline rate first. Extreme baselines (below 5% or above 95%) may require 5-10x more samples than moderate rates for the same absolute effect size. Consider using relative effect sizes (e.g., "10% relative improvement") rather than absolute ones when baseline rates are extreme.
Key Takeaway: Baseline conversion rates near extremes (0% or 100%) require dramatically larger sample sizes than moderate rates due to lower variance — always account for your actual baseline when calculating power.