Course contentsShow
← Machine Learning and Deep Learning
Lesson 3076 of 3,53867. A/B Testing and Deployment StrategiesFree lesson

Variance Reduction Techniques

Apply stratification, CUPED, and regression adjustment to increase statistical power and reduce experiment duration.

Variance Reduction Techniques

You'll learn: How to apply stratification, CUPED, and regression adjustment to make your A/B tests more sensitive and finish faster by reducing noise in your measurements.

The Core Problem

When running A/B tests, high variance in your metrics means you need more data (and more time) to detect real effects. Imagine trying to hear a conversation in a noisy room—the louder the background noise (variance), the harder it is to detect the signal (treatment effect). Variance reduction techniques act like noise-canceling headphones for your experiments.

Three Powerful Techniques

Stratification

Split your users into meaningful groups (strata) before randomization—like separating mobile from desktop users. Analyze each stratum separately, then combine results. This removes between-group variance that would otherwise obscure your treatment effect.

CUPED (Controlled-Experiment Using Pre-Experiment Data)

Use historical data as a "control variate." If a user had high engagement before the experiment, adjust their post-experiment metric accordingly. The formula: adjusted_metric = metric - θ × (pre_metric - mean_pre_metric). This removes user-level variance that existed before treatment.

Think of it like: Measuring weight loss by looking at change from baseline, not absolute weight—much less noisy!

Regression Adjustment

Add covariates (user features) to a regression model predicting your metric, then test the treatment coefficient. Features like user tenure, past behavior, or device type explain variance unrelated to your treatment, making the effect clearer.

Impact

These techniques can reduce required sample size by 20-50%, shortening experiments from weeks to days while maintaining statistical power.

Key Takeaway: Variance reduction techniques leverage stratification, pre-experiment data (CUPED), and regression adjustment to remove noise unrelated to your treatment, dramatically increasing statistical power and reducing experiment duration.