Variance Reduction Techniques
You'll learn: How to apply stratification, CUPED, and regression adjustment to make your A/B tests more sensitive and finish faster by reducing noise in your measurements.
The Core Problem
When running A/B tests, high variance in your metrics means you need more data (and more time) to detect real effects. Imagine trying to hear a conversation in a noisy room—the louder the background noise (variance), the harder it is to detect the signal (treatment effect). Variance reduction techniques act like noise-canceling headphones for your experiments.
Three Powerful Techniques
Stratification
Split your users into meaningful groups (strata) before randomization—like separating mobile from desktop users. Analyze each stratum separately, then combine results. This removes between-group variance that would otherwise obscure your treatment effect.
CUPED (Controlled-Experiment Using Pre-Experiment Data)
Use historical data as a "control variate." If a user had high engagement before the experiment, adjust their post-experiment metric accordingly. The formula: adjusted_metric = metric - θ × (pre_metric - mean_pre_metric). This removes user-level variance that existed before treatment.
Think of it like: Measuring weight loss by looking at change from baseline, not absolute weight—much less noisy!
Regression Adjustment
Add covariates (user features) to a regression model predicting your metric, then test the treatment coefficient. Features like user tenure, past behavior, or device type explain variance unrelated to your treatment, making the effect clearer.
Impact
These techniques can reduce required sample size by 20-50%, shortening experiments from weeks to days while maintaining statistical power.
Key Takeaway: Variance reduction techniques leverage stratification, pre-experiment data (CUPED), and regression adjustment to remove noise unrelated to your treatment, dramatically increasing statistical power and reducing experiment duration.