Effect Size: The Minimum Detectable Effect
You'll learn how to define the smallest difference worth detecting in your A/B test—the practical threshold that makes running the test worthwhile.
What Is the Minimum Detectable Effect?
The Minimum Detectable Effect (MDE) is the smallest change in your success metric that would be meaningful enough to justify implementing the treatment variant. It's not just about statistical detection—it's about practical significance.
Think of it like adjusting your home thermostat. You could detect a 0.1°F difference with a sensitive thermometer, but would you care? Probably not. You only care about changes large enough to actually affect your comfort—say, 2-3°F. Similarly, in A/B testing, the MDE is the smallest improvement that's worth your effort to detect and act upon.
Why MDE Matters
Setting your MDE upfront serves two critical purposes:
-
Sample size calculation: A smaller MDE requires more data to reliably detect. If you want to catch a 0.5% improvement vs. a 5% improvement, you'll need dramatically more users in your test.
-
Business relevance: It forces you to ask: "What size of change would actually move the needle for our business?" A statistically significant 0.1% click-rate increase might not justify engineering resources to deploy the change.
Practical Considerations
Your MDE should balance statistical feasibility with business impact. Consider factors like implementation costs, current baseline performance, and strategic importance. A feature requiring six months of development needs a larger MDE than a simple color change.
Key Takeaway: The Minimum Detectable Effect defines the practical threshold for your A/B test—the smallest improvement worth detecting—and directly determines how much data you'll need to collect.