Course contentsShow
← Machine Learning and Deep Learning
Lesson 3070 of 3,53867. A/B Testing and Deployment StrategiesFree lesson

Statistical Foundations: Hypothesis Testing

Learn null and alternative hypotheses, p-values, significance levels, and statistical power for A/B tests.

Statistical Foundations: Hypothesis Testing

You'll learn the core statistical framework that lets you determine whether observed differences between A/B test variants are real or just random noise.

The Core Idea

When you run an A/B test comparing two ML models, you're asking: "Is Model B genuinely better than Model A, or did I just get lucky?" Hypothesis testing provides the mathematical rigor to answer this question.

Null and Alternative Hypotheses

  • Null Hypothesis (H₀): The default assumption that there's no difference between variants. "Model B performs the same as Model A."
  • Alternative Hypothesis (H₁): What you're trying to prove. "Model B performs better than Model A."

Think of H₀ as "innocent until proven guilty" – you assume no effect exists until evidence proves otherwise.

P-Values

The p-value answers: "If the null hypothesis were true, what's the probability of seeing results this extreme (or more) by random chance?"

  • Low p-value (< 0.05): Strong evidence against H₀
  • High p-value: Weak evidence; you can't reject H₀

Significance Level (α)

Your pre-defined threshold for decision-making, typically α = 0.05 (5%). If p-value < α, you reject H₀ and declare the result "statistically significant." This controls your false positive rate.

Statistical Power (1 - β)

The probability of correctly detecting a real effect when it exists. Higher power (typically aim for 80-90%) means you're less likely to miss genuine improvements. Power depends on:

  • Sample size (more data = more power)
  • Effect size (bigger differences are easier to detect)
  • Significance level

Key Takeaway: Hypothesis testing provides the statistical framework to decide if A/B test differences are signal or noise—p-values measure evidence strength, α controls false positives, and power ensures you can detect real effects.