Course contentsShow
← Data Science
Lesson 456 of 2,14512. ANOVA and Multiple ComparisonsFree lesson

The Multiple Comparisons Problem

How conducting many pairwise tests inflates the familywise error rate and increases false positives.

The Multiple Comparisons Problem

You'll learn why running many statistical tests at once dramatically increases your chances of finding false positives.

The Core Issue

Imagine you conduct an ANOVA and find that at least one group differs from the others. Your natural next question is: "Which groups are different?" You might be tempted to run every possible pairwise t-test between groups.

Here's the problem: each individual test carries its own risk of a Type I error (false positive). If you use α = 0.05, you have a 5% chance of incorrectly rejecting the null hypothesis in one test. But when you run multiple tests, these error rates accumulate.

The Familywise Error Rate

Think of it like this: if you flip a fair coin once, you have a 50% chance of heads. But if you flip it 10 times, you're almost certain to see heads at least once. Similarly, the more pairwise comparisons you make, the higher your chance of finding at least one "significant" result purely by chance.

With 5 groups, there are 10 possible pairwise comparisons. Even if no true differences exist, running all 10 tests at α = 0.05 gives you roughly a 40% chance of falsely detecting at least one "significant" difference somewhere. That's far worse than the 5% you intended!

This inflated error rate across all your comparisons is called the familywise error rate.

Why This Matters

Without correction, you'd report spurious findings—differences that don't actually exist. This is why ANOVA alone isn't enough; you need specialized post-hoc tests that control for multiple comparisons.

Key Takeaway: Running multiple pairwise tests inflates your familywise error rate far beyond α, dramatically increasing false positives—this is why post-hoc correction methods are essential after ANOVA.