Correlated Features Problem
What you'll learn: How correlated features distort importance scores by splitting credit, making interpretation unreliable.
The Core Problem
When two or more features are highly correlated (they move together), importance methods face a credit assignment problem. Imagine you're trying to determine whether studying with coffee or studying with tea makes students perform better—but in your data, almost everyone who drinks coffee also drinks tea.
The model can use either feature to make predictions, since they carry nearly identical information. As a result, the importance gets arbitrarily split between them. One run might attribute 60% importance to coffee and 20% to tea; another might reverse it. Neither tells you the true story: that caffeine intake (the underlying factor) drives performance.
Why This Matters
This problem particularly affects:
- Permutation importance: Shuffling one correlated feature leaves the other intact, so predictions don't degrade much—underestimating both features
- Tree-based importance (MDI): The tree randomly picks which correlated feature to split on first, arbitrarily assigning it higher importance
- Linear coefficients: Multicollinearity inflates variance in coefficient estimates, making them unstable
The danger isn't just incorrect rankings—it's that small data changes can completely reorder your top features, destroying reproducibility.
What You Should Do
Before trusting feature importance, check feature correlations. Group highly correlated features conceptually, or use techniques like PCA to combine them. Recognize that importance scores for redundant features should be interpreted as a group, not individually.
Key Takeaway: Correlated features split importance scores unpredictably, making individual rankings unreliable; always examine feature relationships before interpreting importance.