Course contentsShow
← Data Science
Lesson 624 of 2,14516. Multiple Linear RegressionFree lesson

Multiple Testing Considerations

Understand the implications of testing multiple coefficients and approaches to control family-wise error rate.

Multiple Testing Considerations

You'll learn why running many hypothesis tests at once inflates your risk of false discoveries—and how to protect against it.

When you build a multiple regression model with many predictors, you're likely running t-tests on every coefficient. Each test comes with a probability (typically 5%) of falsely rejecting the null hypothesis—a Type I error. Run enough tests, and you're almost guaranteed to get "significant" results purely by chance.

The Multiple Testing Problem

Imagine flipping 20 coins. Even fair coins, you'd expect about one to land heads 10 times out of 10 just by luck. Similarly, if you test 20 coefficients at α = 0.05, you'd expect one "significant" result even if none of your predictors truly matter.

This is called inflating the family-wise error rate (FWER)—the probability of making at least one false discovery across all your tests increases dramatically with each additional test.

Controlling the Family-Wise Error Rate

To maintain your desired error rate across multiple tests, you can:

  • Bonferroni correction: Divide your significance level by the number of tests. Testing 10 coefficients? Use α = 0.05/10 = 0.005 for each test. Conservative but simple.

  • Use the global F-test first: Before examining individual coefficients, check if any predictors matter. If the overall F-test fails, individual t-tests are less trustworthy.

  • Be selective: Test only coefficients you care about based on theory, not all of them exploratorily.

Think of it like a prosecutor examining 100 suspects: if you use weak evidence standards, you'll wrongly accuse several innocent people. Tighten your standards when casting a wide net.

Key Takeaway: Testing multiple coefficients increases your chance of false positives; control the family-wise error rate using corrections like Bonferroni or rely on the global F-test to guide interpretation.