Running Underpowered Tests
You'll learn why launching an A/B test without enough sample size wastes time and often leads to completely missing real, actionable effects.
The Core Problem
Remember that statistical power is the probability of detecting an effect when it actually exists. Running an underpowered test means your sample size is too small relative to the minimum detectable effect you care about—typically yielding power below 80%.
When you run an underpowered A/B test, two bad things happen:
-
You miss real effects – Even if your new feature genuinely improves conversion by 2%, your test might conclude "no significant difference" simply because you didn't collect enough data. This is a false negative (Type II error).
-
You waste resources – Your engineering team built the feature, you split traffic for weeks, analyzed results... only to end with "inconclusive." The opportunity cost is enormous.
The Hidden Danger
Underpowered tests are insidious because they feel scientific. You randomized users, waited patiently, ran a proper t-test—but the experiment was doomed from the start. It's like trying to hear a whisper in a loud room: the signal exists, but your setup can't capture it.
Why This Happens
Teams often run underpowered tests due to:
- Impatience ("let's just launch and see what happens")
- Underestimating the required sample size
- Testing effects smaller than realistic power calculations support
- Business pressure to decide quickly
Key Takeaway: An underpowered test is worse than no test at all—it burns resources while providing false confidence in "no effect" conclusions. Always validate your sample size meets your power requirements before launching.