Hypothesis Testing Framework
What you'll learn: You'll learn how to set up and interpret a formal statistical test to decide whether observed data provides strong enough evidence to reject a baseline assumption.
The Core Idea
Think of hypothesis testing like a courtroom trial. We start by assuming innocence (our null hypothesis, often written as H₀), then ask: "Is the evidence strong enough to reject this assumption?"
In statistics, the null hypothesis (H₀) represents the status quo or "no effect" claim. For example: "This new ML model performs no better than random guessing" or "The mean accuracy is 50%."
The alternative hypothesis (H₁ or Hₐ) is what you're trying to prove: "The model does better than random" or "The mean accuracy is greater than 50%."
P-Values and Significance Levels
After collecting data, we calculate a p-value: the probability of seeing results as extreme as ours if the null hypothesis were true.
- A small p-value (like 0.01) means: "Wow, if H₀ were true, this data would be really unlikely. Maybe H₀ is wrong!"
- A large p-value (like 0.60) means: "This data is pretty consistent with H₀. No reason to reject it."
The significance level (α) is your decision threshold, commonly set at 0.05. If p-value < α, you reject H₀ and accept H₁. If p-value ≥ α, you fail to reject H₀ (note: you don't "accept" it, you simply lack evidence against it).
Analogy
The p-value is like asking: "If my friend really is honest (H₀), what are the chances they'd 'accidentally' guess my password on the first try?" If that chance is tiny, you doubt their honesty claim.
Key Takeaway: Hypothesis testing provides a structured framework to evaluate claims using data: formulate H₀ and H₁, compute a p-value from your sample, and compare it to your significance level α to make a principled decision.