Testing Too Many Variants
You'll learn: How splitting your sample across too many variants dilutes statistical power and inflates false positive risk unless you apply multiple-testing corrections.
The Power Dilution Problem
When you test many variants (A vs B vs C vs D vs E...), you're splitting your fixed sample size across all groups. Each comparison gets fewer observations than if you'd just tested A vs B alone.
The consequences:
- Reduced power per test: With the same overall sample size, each pairwise comparison has less data and thus less ability to detect real effects
- Increased family-wise error rate (FWER): More comparisons mean more chances to find false positives by chance alone
The Math Behind the Issue
Imagine you designed for 80% power with two variants. Now you split into five variants. Each gets ~40% of the original per-group sample size. Power drops significantly—often below 50%—for detecting the same effect size.
Meanwhile, if you run multiple pairwise tests (A vs B, A vs C, B vs C...), your false positive rate compounds. With 5 variants and no correction, you might run 10 comparisons. At α = 0.05 each, your chance of at least one false positive jumps to nearly 40%, far exceeding the nominal 5% FWER.
The Solution
Apply corrections from earlier lessons: Bonferroni, Holm-Bonferroni, or FDR control. Better yet, limit variants to 2-3 well-justified alternatives. Pre-register your comparisons to avoid data-driven fishing expeditions.
Key Takeaway: Testing too many variants simultaneously dilutes statistical power in each comparison and dramatically increases the risk of false discoveries unless you use strict multiple-testing corrections and increase sample size accordingly.