Power Analysis Fundamentals
You'll learn what statistical power means and why it's essential for designing reliable A/B tests.
What is Power?
In A/B testing, power (represented as 1-β, where β is the probability of a Type II error) is the probability that your test will correctly detect a true effect when it actually exists.
Think of it this way: imagine you're a metal detector operator searching for buried treasure. Power is your detector's ability to beep when there really is treasure underground. A high-power detector rarely misses real treasure; a low-power detector might walk right over it without alerting you.
Understanding Power in Practice
When you run an A/B test and there's a genuine difference between your control and treatment groups, power tells you: "What's the chance I'll actually detect this difference as statistically significant?"
- High power (e.g., 0.80 or 80%): You have an 80% chance of detecting a real effect
- Low power (e.g., 0.30 or 30%): You'll miss real effects 70% of the time—your test is unreliable
Why Power Matters
A test with low power is wasteful. You might invest time and resources running an experiment, only to conclude "no significant difference" when a real effect exists—you just didn't have enough sensitivity to find it. This is called a false negative or Type II error.
Standard practice aims for 80% power, meaning you accept a 20% chance of missing a true effect.
Key Takeaway: Power (1-β) quantifies your test's ability to detect real effects—higher power means fewer missed discoveries and more reliable A/B testing decisions.