Course contentsShow
← Mathematics
Lesson 3081 of 3,12160. Statistics — ModelingFree lesson

Variable Selection Methods

Learn forward selection, backward elimination, and stepwise regression to choose an optimal subset of predictors.

Variable Selection Methods

You'll learn how to systematically choose the best predictors for your regression model using forward selection, backward elimination, and stepwise regression.

The Problem: Too Many Predictors

In multiple regression, you often face dozens of potential predictor variables. Including all of them can lead to overfitting, multicollinearity, and models that are hard to interpret. Variable selection methods help you find an optimal subset that balances predictive power with model simplicity.

Three Core Approaches

Forward Selection

Start with no predictors. At each step, add the variable that most improves your model (typically the one with the lowest p-value or highest contribution to R²). Stop when adding variables no longer significantly improves the model.

Analogy: Building a team by recruiting one person at a time—always picking whoever adds the most value.

Backward Elimination

Start with all predictors. At each step, remove the least significant variable (highest p-value). Continue until all remaining variables meet your significance threshold (e.g., p < 0.05).

Analogy: Trimming a roster by cutting the weakest player each round until everyone earns their spot.

Stepwise Regression

Combines both approaches. At each step, you can either add the best new variable OR remove a variable that's become non-significant. This handles situations where adding a new predictor makes an old one redundant.

Analogy: A flexible team manager who can both recruit and release players as the lineup evolves.

Choosing Your Criteria

Common stopping rules include significance thresholds (p-values), information criteria (AIC, BIC—though these haven't been formally introduced yet), or adjusted R² changes. Each method has tradeoffs in computational cost and risk of missing interactions.

Key Takeaway: Variable selection methods automate choosing predictor subsets—forward selection builds up, backward elimination trims down, and stepwise does both—helping you balance model fit with simplicity.