Power Analysis in Study Design
What you'll learn: How to use power analysis to calculate the sample size you need before you collect any data.
Why Plan Sample Size in Advance?
Imagine you're planning a road trip and need to know how much gas to buy. Too little, and you won't reach your destination. Too much, and you've wasted money. Sample size works the same way: collect too few observations and you won't detect real effects (low power); collect too many and you've wasted time and resources.
Power analysis is the process of determining the minimum sample size required to detect an effect of a given size with adequate statistical power, all while controlling your Type I error rate (alpha).
The Four Connected Pieces
Power analysis balances four quantities:
- Sample size (n) – what you're solving for
- Effect size – how large a difference you expect to find
- Alpha (α) – your significance level (usually 0.05)
- Power (1 - β) – your desired probability of detecting the effect (usually 0.80)
When you specify any three of these, you can calculate the fourth. In study design, you typically specify effect size, alpha, and desired power, then calculate the required sample size.
Practical Application
Before running an A/B test on conversion rates, you'd ask: "If the true conversion lift is 2 percentage points, how many users do I need to observe to have 80% power to detect it at α = 0.05?" Power analysis gives you that number, preventing under-powered studies that waste resources or over-powered studies that cost too much.
Key Takeaway: Power analysis turns hypothesis testing from a post-hoc "did we find something?" question into a proactive planning tool—helping you design studies that are both efficient and capable of detecting the effects you care about.