Effect Size and Practical Significance in t-Tests
You'll learn how to measure whether a statistically significant difference actually matters in the real world.
The Problem with P-Values Alone
A t-test can tell you whether two means are statistically different—whether the difference is unlikely due to chance. But with a large enough sample, even a tiny, trivial difference can become "statistically significant." For example, a drug might lower blood pressure by 0.5 mmHg (p < 0.05)—statistically significant, but medically meaningless.
Practical significance asks: How large is the effect, and does it matter?
Cohen's d: Standardizing the Difference
Cohen's d measures effect size by expressing the difference between two means in standard deviation units:
d = (Mean₁ - Mean₂) / Pooled Standard Deviation
This standardization makes the effect size comparable across studies, regardless of measurement scale.
Interpreting Cohen's d
- |d| ≈ 0.2: Small effect (like the difference in height between 15- and 16-year-old girls)
- |d| ≈ 0.5: Medium effect (easily noticeable in practice)
- |d| ≈ 0.8: Large effect (substantial, obvious difference)
Why Effect Size Matters
Think of it this way: a p-value answers "Is there a difference?" while effect size answers "How big is that difference, and should I care?"
A medication study might find p = 0.001 (highly significant) but d = 0.1 (tiny effect)—statisticians celebrate, but doctors shrug. Conversely, d = 0.9 but p = 0.08 (not significant) might suggest a promising treatment that needs a larger sample to confirm.
Key Takeaway: Always report effect sizes like Cohen's d alongside p-values; statistical significance tells you if an effect exists, but effect size tells you if it's meaningful enough to matter in practice.