Pre-Registration and Correction Strategy
What you'll learn: How to lock in your testing plan upfront to prevent accidental (or intentional) p-hacking and preserve the integrity of your A/B test results.
Why Pre-Register Your A/B Test?
Imagine you're a teacher who decides which questions to grade after seeing student answers. That's essentially what happens when you run an A/B test without documenting your plan first. Pre-registration means writing down your hypotheses, metrics, sample size, stopping rules, and correction methods before you peek at any results.
This practice prevents p-hacking—the tendency to keep testing variations or slicing data until something looks significant. Without pre-registration, you might unconsciously choose the most favorable multiple testing correction (or skip it entirely), turning your rigorous experiment into a fishing expedition.
What to Document Before Running Your Test
Your pre-registration should include:
- Primary and secondary metrics you'll measure
- Hypotheses you're testing (which variations, what direction)
- Sample size calculation and minimum detectable effect
- Multiple testing correction method (Bonferroni, Holm-Bonferroni, Benjamini-Hochberg, etc.)
- Stopping rules (fixed horizon? sequential testing plan?)
- Subgroup analyses you plan to run, if any
Think of it like a recipe: you wouldn't change the ingredients halfway through baking and still claim you followed the recipe.
Choosing Your Correction Strategy
Based on your needs from the previous lessons:
- Testing 3 variations? Document whether you'll use Bonferroni (conservative, control FWER) or Benjamini-Hochberg (exploratory, control FDR)
- Running 20 metrics? Decide which is primary and how secondary metrics will be corrected
Key Takeaway: Pre-registering your A/B test design and correction strategy before collecting data is your best defense against false discoveries and maintains the statistical validity of your conclusions.