Course contentsShow
← Data Science
Lesson 329 of 2,1459. Hypothesis Testing FoundationsFree lesson

Choosing α Before Analysis

Why setting the significance level before seeing data prevents p-hacking and ensures valid inference.

Choosing α Before Analysis

You'll learn why committing to your significance level before you look at the data is essential for honest, reproducible science.

The Pre-Registration Principle

Imagine a basketball player who takes a shot, watches where the ball lands, then paints the target around it afterward. They'd hit "bullseye" every time! Similarly, if you set your significance level (α) after seeing your p-value, you can always make your result look significant.

Choosing α before analysis means deciding your threshold for rejecting the null hypothesis—typically 0.05, 0.01, or 0.10—before you collect or analyze your data. This pre-commitment prevents you from unconsciously (or deliberately) moving the goalposts to get the answer you want.

Why This Matters: Avoiding P-Hacking

When researchers peek at results and then adjust their α, they engage in p-hacking—manipulating the analysis until they get p < 0.05. This might mean:

  • Lowering α to 0.10 when p = 0.08 ("close enough!")
  • Raising α requirements when results are too significant ("let's be more conservative")
  • Running multiple tests and reporting only those that "work"

Pre-specifying α keeps you honest. You set your risk tolerance for Type I errors (false positives) based on the context—medical trials might use α = 0.01 for safety, while exploratory marketing tests might accept α = 0.10—not based on what makes your current data look good.

Valid Inference Requires Pre-Commitment

The entire logic of hypothesis testing assumes you defined your decision rule before observing the outcome. Your p-value only has its stated meaning if α was fixed in advance. Otherwise, you're invalidating the probability calculations that make inference possible.

Key Takeaway: Always choose and document your significance level α before analyzing data—it's the cornerstone of valid statistical inference and protects against p-hacking.