Statistical Foundations: Hypothesis Testing
You'll learn the core statistical framework that lets you determine whether observed differences between A/B test variants are real or just random noise.
The Core Idea
When you run an A/B test comparing two ML models, you're asking: "Is Model B genuinely better than Model A, or did I just get lucky?" Hypothesis testing provides the mathematical rigor to answer this question.
Null and Alternative Hypotheses
- Null Hypothesis (H₀): The default assumption that there's no difference between variants. "Model B performs the same as Model A."
- Alternative Hypothesis (H₁): What you're trying to prove. "Model B performs better than Model A."
Think of H₀ as "innocent until proven guilty" – you assume no effect exists until evidence proves otherwise.
P-Values
The p-value answers: "If the null hypothesis were true, what's the probability of seeing results this extreme (or more) by random chance?"
- Low p-value (< 0.05): Strong evidence against H₀
- High p-value: Weak evidence; you can't reject H₀
Significance Level (α)
Your pre-defined threshold for decision-making, typically α = 0.05 (5%). If p-value < α, you reject H₀ and declare the result "statistically significant." This controls your false positive rate.
Statistical Power (1 - β)
The probability of correctly detecting a real effect when it exists. Higher power (typically aim for 80-90%) means you're less likely to miss genuine improvements. Power depends on:
- Sample size (more data = more power)
- Effect size (bigger differences are easier to detect)
- Significance level
Key Takeaway: Hypothesis testing provides the statistical framework to decide if A/B test differences are signal or noise—p-values measure evidence strength, α controls false positives, and power ensures you can detect real effects.