Course contentsShow
← Data Science
Lesson 1508 of 2,14537. A/B TestingFree lesson

Pre-Registration and Correction Strategy

Choosing and documenting correction methods before running experiments to maintain statistical validity and avoid p-hacking.

Pre-Registration and Correction Strategy

What you'll learn: How to lock in your testing plan upfront to prevent accidental (or intentional) p-hacking and preserve the integrity of your A/B test results.

Why Pre-Register Your A/B Test?

Imagine you're a teacher who decides which questions to grade after seeing student answers. That's essentially what happens when you run an A/B test without documenting your plan first. Pre-registration means writing down your hypotheses, metrics, sample size, stopping rules, and correction methods before you peek at any results.

This practice prevents p-hacking—the tendency to keep testing variations or slicing data until something looks significant. Without pre-registration, you might unconsciously choose the most favorable multiple testing correction (or skip it entirely), turning your rigorous experiment into a fishing expedition.

What to Document Before Running Your Test

Your pre-registration should include:

  • Primary and secondary metrics you'll measure
  • Hypotheses you're testing (which variations, what direction)
  • Sample size calculation and minimum detectable effect
  • Multiple testing correction method (Bonferroni, Holm-Bonferroni, Benjamini-Hochberg, etc.)
  • Stopping rules (fixed horizon? sequential testing plan?)
  • Subgroup analyses you plan to run, if any

Think of it like a recipe: you wouldn't change the ingredients halfway through baking and still claim you followed the recipe.

Choosing Your Correction Strategy

Based on your needs from the previous lessons:

  • Testing 3 variations? Document whether you'll use Bonferroni (conservative, control FWER) or Benjamini-Hochberg (exploratory, control FDR)
  • Running 20 metrics? Decide which is primary and how secondary metrics will be corrected

Key Takeaway: Pre-registering your A/B test design and correction strategy before collecting data is your best defense against false discoveries and maintains the statistical validity of your conclusions.