Documentation and Pre-Registration
You'll learn why documenting your A/B test plan before launch protects against bias and ensures credible results.
The Core Idea
Pre-registration means writing down your entire experiment plan—hypotheses, metrics, sample size, analysis methods—before you collect any data. Think of it like a scientist publishing their research protocol before running an experiment. This "locks in" your decisions so you can't accidentally cherry-pick results that look good after the fact.
Documentation serves as your experimental contract. It prevents HARKing (Hypothesizing After Results are Known), where you retrofit explanations to unexpected patterns. It also protects against p-hacking—trying multiple analyses until something looks significant.
What to Document
Your pre-registration should include:
- Hypothesis: The specific change and expected directional effect
- Primary metric: Your one key success measure (e.g., conversion rate)
- Secondary metrics: Supporting metrics and guardrail metrics to monitor
- Sample size calculation: Based on your Minimum Detectable Effect and power
- Randomization unit: User, session, or other unit you defined
- Duration: How long the test will run
- Analysis plan: Statistical test you'll use, significance level (usually α = 0.05), how you'll handle missing data
Why It Matters
Imagine flipping a coin 100 times, then claiming you predicted the exact sequence. Sounds absurd, right? But without pre-registration, you could run dozens of "post-hoc" analyses and present the one significant result as if you planned it all along. Pre-registration adds scientific rigor and makes your findings trustworthy to stakeholders.
Key Takeaway: Pre-registering your A/B test creates an audit trail that ensures your analysis stays objective and your conclusions remain credible—document everything before you see the data.