Sample Size Requirements
You'll learn how to verify that your sample is large enough for a proportion test's normal approximation to be valid.
Why Sample Size Matters for Proportion Tests
When testing proportions (like "What percentage of users clicked the ad?"), we rely on a mathematical shortcut: approximating the binomial distribution with the normal distribution. This makes calculations easier and P-values accessible. But this approximation only works well when you have enough data.
The Success-Failure Condition
Before conducting a proportion test, you must check the success-failure condition:
- Number of successes ≥ 10
- Number of failures ≥ 10
Where:
- Successes = n × p̂ (sample size times observed proportion)
- Failures = n × (1 - p̂)
Analogy: Think of approximating a bumpy staircase with a smooth ramp. If you only have 3 or 4 steps, the ramp looks nothing like the stairs. With 50 steps, the ramp becomes a pretty good stand-in. The success-failure condition ensures your "staircase" has enough "steps" for the normal curve to fit well.
For Two-Sample Tests
Check the condition for both groups separately:
- Group 1: n₁p̂₁ ≥ 10 and n₁(1-p̂₁) ≥ 10
- Group 2: n₂p̂₂ ≥ 10 and n₂(1-p̂₂) ≥ 10
If any group fails this check, your normal approximation may be unreliable, leading to incorrect P-values and bad decisions.
What If You Fail the Check?
Use exact methods (like Fisher's exact test) or collect more data. Never ignore this condition—it protects you from misleading conclusions.
Key Takeaway: Always verify the success-failure condition (≥10 successes and ≥10 failures) before trusting proportion test results based on normal approximation.