Course contentsShow
← Data Science
Lesson 585 of 2,14515. Regression DiagnosticsFree lesson

Remedies: Variable Selection

Address multicollinearity by removing redundant predictors or combining correlated variables into composites.

Remedies: Variable Selection

You'll learn how to fix multicollinearity by strategically choosing which predictors to keep in your model.

The Core Idea

When you've detected multicollinearity (high VIF values, unstable coefficients, inflated standard errors), one of the simplest remedies is variable selection—deliberately removing redundant predictors that contribute overlapping information. Think of it like decluttering a workspace: if two tools do the same job, keeping both just creates confusion.

How Variable Selection Works

When predictors are highly correlated, they're competing to explain the same variance in your outcome. By removing one (or more) of the redundant variables, you:

  • Reduce multicollinearity: VIF values drop for remaining predictors
  • Stabilize coefficient estimates: Less wobbling between models
  • Improve interpretability: Clearer story about what drives your outcome
  • Maintain predictive power: Often little loss in R-squared

Which Variable to Remove?

Use these strategies:

  1. Correlation matrix examination: Drop one variable from pairs with correlation > 0.8-0.9
  2. VIF-guided removal: Remove the predictor with highest VIF, recalculate, repeat
  3. Domain knowledge: Keep variables that make theoretical sense; drop proxies or duplicates
  4. Model comparison: Test models with different subsets and compare fit metrics

Combining Variables

Instead of dropping, sometimes create composite variables by averaging or summing highly correlated predictors. For example, if you have height_cm and height_inches (perfectly correlated), just pick one—or if measuring overlapping concepts like multiple test scores, create an average "academic_performance" variable.

Key Takeaway: Variable selection resolves multicollinearity by removing or combining redundant predictors, simplifying your model while preserving its explanatory power.