Course contentsShow
← Data Science
Lesson 1485 of 2,14537. A/B TestingFree lesson

Documentation and Pre-Registration

Creating experiment documentation including hypothesis, metrics, and analysis plan before launch.

Documentation and Pre-Registration

You'll learn why documenting your A/B test plan before launch protects against bias and ensures credible results.

The Core Idea

Pre-registration means writing down your entire experiment plan—hypotheses, metrics, sample size, analysis methods—before you collect any data. Think of it like a scientist publishing their research protocol before running an experiment. This "locks in" your decisions so you can't accidentally cherry-pick results that look good after the fact.

Documentation serves as your experimental contract. It prevents HARKing (Hypothesizing After Results are Known), where you retrofit explanations to unexpected patterns. It also protects against p-hacking—trying multiple analyses until something looks significant.

What to Document

Your pre-registration should include:

  • Hypothesis: The specific change and expected directional effect
  • Primary metric: Your one key success measure (e.g., conversion rate)
  • Secondary metrics: Supporting metrics and guardrail metrics to monitor
  • Sample size calculation: Based on your Minimum Detectable Effect and power
  • Randomization unit: User, session, or other unit you defined
  • Duration: How long the test will run
  • Analysis plan: Statistical test you'll use, significance level (usually α = 0.05), how you'll handle missing data

Why It Matters

Imagine flipping a coin 100 times, then claiming you predicted the exact sequence. Sounds absurd, right? But without pre-registration, you could run dozens of "post-hoc" analyses and present the one significant result as if you planned it all along. Pre-registration adds scientific rigor and makes your findings trustworthy to stakeholders.

Key Takeaway: Pre-registering your A/B test creates an audit trail that ensures your analysis stays objective and your conclusions remain credible—document everything before you see the data.