Course contentsShow
← Data Science
Lesson 1749 of 2,14541. Marketing and Growth AnalyticsFree lesson

Measuring Statistical Significance

Apply hypothesis tests to determine if observed lift in treatment group is statistically significant.

Measuring Statistical Significance

You'll learn how to determine whether an observed lift in your treatment group represents a real effect or just random chance.

Why Statistical Significance Matters

You've run a geo-lift experiment or holdout test and observed that the treatment group outperformed the control by 8%. Great news, right? Not so fast—that difference could be due to random variation rather than your marketing intervention. Statistical significance testing tells you whether the observed lift is convincing enough to act on.

The Core Idea

A hypothesis test starts with a null hypothesis (H₀): "There is no real difference between treatment and control groups." You then calculate the probability of seeing your observed results (or more extreme) if H₀ were true. This probability is called the p-value.

If the p-value falls below your significance threshold (typically α = 0.05), you reject the null hypothesis and conclude the lift is statistically significant—meaning it's unlikely to be random noise.

Think of it like a courtroom: you assume innocence (no effect) until the evidence is strong enough to convict (conclude there's a real effect).

Common Tests for Incrementality

  • T-test: Compares means between two groups (treatment vs control)
  • Z-test: Used when you have large samples or known population variance
  • Proportion test: When your metric is a conversion rate or percentage

You'll also examine confidence intervals: if the 95% confidence interval for your lift doesn't include zero, you have statistical significance at α = 0.05.

Key Considerations

Watch for sample size—small experiments may lack statistical power to detect real effects. And remember: statistical significance ≠ practical significance. A 0.1% lift might be significant but not worth the investment.

Key Takeaway: Statistical significance testing uses hypothesis tests (like t-tests) to confirm that observed lift in treatment groups reflects true incrementality rather than random variation, typically requiring p < 0.05 before declaring results actionable.