Setting Alpha: Choosing Your Significance Level
You'll learn how to pick the right significance level by weighing the costs of false positives versus false negatives.
Why Alpha Isn't One-Size-Fits-All
You've seen that α = 0.05 is standard, but that's just a convention—not a universal rule. The "right" significance level depends on what's at stake when you make the wrong decision.
The Cost-Benefit Framework
Think of choosing alpha like setting a smoke detector's sensitivity:
-
Lower alpha (0.01): Like a less-sensitive detector. You reduce false alarms (Type I errors) but risk missing real fires (Type II errors increase). Use when false positives are expensive or damaging.
-
Higher alpha (0.10): Like a more-sensitive detector. You catch more real fires but get more false alarms. Use when missing a true effect is more costly than a false positive.
Real-World Examples
Medical screening (α = 0.10): Missing a disease (Type II) could be fatal, so you accept more false positives to catch real cases early.
Criminal trials (α = 0.01): Convicting an innocent person (Type I) is devastating, so you demand very strong evidence before rejecting innocence.
Marketing A/B test (α = 0.05): Balanced stakes—you want reasonable confidence without being overly cautious.
The Decision Process
- Ask: What happens if I reject H₀ when it's actually true?
- Ask: What happens if I fail to reject H₀ when it's actually false?
- Compare costs: Which error would cause more harm?
- Set alpha accordingly: Lower if Type I is costly; higher if Type II is costly.
Key Takeaway: Choose your significance level before collecting data by assessing the relative costs of Type I and Type II errors in your specific context—don't just default to 0.05.