Course contentsShow
← Data Science
Lesson 395 of 2,14510. Hypothesis Tests for MeansFree lesson

Permutation Tests for Means and Beyond

Use resampling to generate the null distribution of any test statistic without parametric assumptions.

Permutation Tests for Means and Beyond

You'll learn how to test hypotheses without assuming normal distributions by reshuffling your data to create the null distribution.

The Core Idea

A permutation test (also called a randomization test) uses resampling to generate the null distribution of any test statistic—not just means. Instead of relying on mathematical theory (like the t-distribution), you shuffle your data randomly to simulate what would happen if the null hypothesis were true.

Think of it like this: If there's truly no difference between two groups, then the group labels are arbitrary. You could randomly reassign observations to groups, recalculate your test statistic, and repeat thousands of times. This creates an empirical distribution of what your test statistic looks like under the null hypothesis.

How It Works

  1. Calculate your observed test statistic (e.g., difference in means, median difference, or anything else)
  2. Pool all observations and randomly reassign them to groups
  3. Recalculate the test statistic for this permuted dataset
  4. Repeat steps 2-3 thousands of times (e.g., 10,000 permutations)
  5. Compare your observed statistic to this permutation distribution
  6. Calculate the p-value as the proportion of permuted statistics as extreme or more extreme than your observed value

Why This Matters

Permutation tests don't require normality, equal variances, or large samples. They work with any test statistic you can compute—means, medians, variances, correlations, or custom metrics. This makes them incredibly flexible when parametric assumptions fail or when you're testing something unusual.

Key Takeaway: Permutation tests generate the null distribution through random shuffling rather than mathematical assumptions, giving you a robust, assumption-free way to test hypotheses about any statistic you can calculate.