Course contentsShow
← Data Science
Lesson 1495 of 2,14537. A/B TestingFree lesson

Power Analysis Fundamentals

Learn how power (1-β) represents the probability of detecting a true effect when it exists.

Power Analysis Fundamentals

You'll learn what statistical power means and why it's essential for designing reliable A/B tests.

What is Power?

In A/B testing, power (represented as 1-β, where β is the probability of a Type II error) is the probability that your test will correctly detect a true effect when it actually exists.

Think of it this way: imagine you're a metal detector operator searching for buried treasure. Power is your detector's ability to beep when there really is treasure underground. A high-power detector rarely misses real treasure; a low-power detector might walk right over it without alerting you.

Understanding Power in Practice

When you run an A/B test and there's a genuine difference between your control and treatment groups, power tells you: "What's the chance I'll actually detect this difference as statistically significant?"

  • High power (e.g., 0.80 or 80%): You have an 80% chance of detecting a real effect
  • Low power (e.g., 0.30 or 30%): You'll miss real effects 70% of the time—your test is unreliable

Why Power Matters

A test with low power is wasteful. You might invest time and resources running an experiment, only to conclude "no significant difference" when a real effect exists—you just didn't have enough sensitivity to find it. This is called a false negative or Type II error.

Standard practice aims for 80% power, meaning you accept a 20% chance of missing a true effect.

Key Takeaway: Power (1-β) quantifies your test's ability to detect real effects—higher power means fewer missed discoveries and more reliable A/B testing decisions.