Minimum Detectable Effect (MDE)
What you'll learn: How to determine the smallest change in your metric that's worth detecting in an A/B test, balancing statistical power with business relevance.
The Core Idea
After you've formulated your hypothesis and chosen your success metric, you need to decide: how small of a change matters? The Minimum Detectable Effect (MDE) is the smallest effect size your A/B test is designed to reliably detect. It's not what you hope to find—it's the threshold below which a change wouldn't be worth the effort of implementing.
Think of it like a metal detector: you could calibrate it to beep at every tiny fleck of aluminum foil, but you'd waste time digging. Instead, you set it to ignore anything smaller than a coin. Similarly, MDE sets your test's sensitivity to ignore noise and focus on changes that actually matter to your business.
Why MDE Matters
MDE directly impacts your sample size requirements. A smaller MDE (detecting tiny changes) requires exponentially more data and longer test duration. A larger MDE needs fewer samples but might miss moderate improvements.
Setting Your MDE
Consider two factors:
- Business impact: Would a 0.5% conversion rate lift justify the engineering cost? Or do you need at least 2%?
- Practical significance: Even if statistically significant, is the change meaningful to users and revenue?
For example, if your current conversion rate is 10%, you might set MDE at 1 percentage point (10% relative increase). Anything smaller won't move the needle enough to matter.
Key Takeaway: MDE defines the smallest effect worth detecting in your test, balancing statistical feasibility with business relevance—choose it based on what changes would actually be worth implementing, not just what's statistically detectable.