Effect Size and Practical Significance
You'll learn how to interpret whether a statistically significant difference in proportions actually matters in practice.
Beyond "Statistically Significant"
You've learned how to test whether a difference between two proportions is statistically significant—meaning it's unlikely due to random chance. But statistical significance alone doesn't tell you if the difference is important or meaningful.
Effect size measures the magnitude of the difference between proportions. Even a tiny difference can be statistically significant with a large enough sample size, but that doesn't mean it's worth caring about.
The Gap Between Significance and Substance
Imagine testing two website signup buttons:
- Button A: 50.1% signup rate
- Button B: 50.0% signup rate
With 1 million visitors, this 0.1% difference might be statistically significant (p < 0.05). But is it practically significant? Probably not—the business impact is negligible.
Compare that to:
- Treatment A: 75% recovery rate
- Treatment B: 50% recovery rate
A 25 percentage-point difference is both statistically and practically significant—it meaningfully changes patient outcomes.
Measuring Effect Size for Proportions
For differences in proportions, the effect size is often just the absolute difference: |p₁ - p₂|. You can also express it as a relative difference or risk ratio depending on context.
Ask yourself:
- Does this difference matter to stakeholders?
- Is the effect large enough to justify action or cost?
- What's the real-world impact of this change?
Key Takeaway: Statistical significance tells you a difference is real; effect size and practical significance tell you whether it matters. Always report and interpret both.