Variable Selection Methods
You'll learn how to systematically choose the best predictors for your regression model using forward selection, backward elimination, and stepwise regression.
The Problem: Too Many Predictors
In multiple regression, you often face dozens of potential predictor variables. Including all of them can lead to overfitting, multicollinearity, and models that are hard to interpret. Variable selection methods help you find an optimal subset that balances predictive power with model simplicity.
Three Core Approaches
Forward Selection
Start with no predictors. At each step, add the variable that most improves your model (typically the one with the lowest p-value or highest contribution to R²). Stop when adding variables no longer significantly improves the model.
Analogy: Building a team by recruiting one person at a time—always picking whoever adds the most value.
Backward Elimination
Start with all predictors. At each step, remove the least significant variable (highest p-value). Continue until all remaining variables meet your significance threshold (e.g., p < 0.05).
Analogy: Trimming a roster by cutting the weakest player each round until everyone earns their spot.
Stepwise Regression
Combines both approaches. At each step, you can either add the best new variable OR remove a variable that's become non-significant. This handles situations where adding a new predictor makes an old one redundant.
Analogy: A flexible team manager who can both recruit and release players as the lineup evolves.
Choosing Your Criteria
Common stopping rules include significance thresholds (p-values), information criteria (AIC, BIC—though these haven't been formally introduced yet), or adjusted R² changes. Each method has tradeoffs in computational cost and risk of missing interactions.
Key Takeaway: Variable selection methods automate choosing predictor subsets—forward selection builds up, backward elimination trims down, and stepwise does both—helping you balance model fit with simplicity.