Applications: Feature Selection and Multicollinearity
What You'll Learn
You'll learn how correlation matrices and partial correlations help you identify redundant predictors and choose better variables for modeling.
Why This Matters
When building predictive models, you want features (predictors) that are informative but not redundant. Two problems arise when predictors are highly correlated with each other:
- Multicollinearity: When predictors are too similar, your model becomes unstable and hard to interpret
- Wasted effort: Including redundant features adds complexity without improving predictions
Think of it like hiring a team: if two people have identical skills, you're not gaining extra capability—you're just creating confusion about who's responsible for what.
Using Correlation Matrices for Feature Selection
A correlation matrix shows you all pairwise correlations between variables at once. Look for:
- High correlations between predictors (e.g., |r| > 0.8): These suggest redundancy
- Low correlations with the outcome: These predictors may not be useful
When two predictors are highly correlated, you can often drop one without losing predictive power.
Using Partial Correlations to Dig Deeper
Sometimes two variables appear correlated only because they both depend on a third variable (a confounding variable you learned about earlier). Partial correlation removes this illusion by controlling for other variables.
This helps you identify which predictors have unique relationships with your outcome—those are the keepers!
Practical Strategy
- Build a correlation matrix of all predictors
- Identify pairs with |r| > 0.8 (strong multicollinearity)
- Use partial correlations to see which variables add unique information
- Drop redundant predictors, keeping those with stronger unique relationships to your outcome
Key Takeaway: Correlation matrices reveal redundant predictors through high pairwise correlations, while partial correlations help you identify which variables contribute unique information—both are essential tools for selecting informative, non-redundant features in modeling.