Course contentsShow
← Machine Learning and Deep Learning
Lesson 3191 of 3,53870. Interpretability: Feature-Based MethodsFree lesson

Correlated Features Problem

Understand how feature correlation affects importance attribution and why redundant features can split importance scores.

Correlated Features Problem

What you'll learn: How correlated features distort importance scores by splitting credit, making interpretation unreliable.

The Core Problem

When two or more features are highly correlated (they move together), importance methods face a credit assignment problem. Imagine you're trying to determine whether studying with coffee or studying with tea makes students perform better—but in your data, almost everyone who drinks coffee also drinks tea.

The model can use either feature to make predictions, since they carry nearly identical information. As a result, the importance gets arbitrarily split between them. One run might attribute 60% importance to coffee and 20% to tea; another might reverse it. Neither tells you the true story: that caffeine intake (the underlying factor) drives performance.

Why This Matters

This problem particularly affects:

  • Permutation importance: Shuffling one correlated feature leaves the other intact, so predictions don't degrade much—underestimating both features
  • Tree-based importance (MDI): The tree randomly picks which correlated feature to split on first, arbitrarily assigning it higher importance
  • Linear coefficients: Multicollinearity inflates variance in coefficient estimates, making them unstable

The danger isn't just incorrect rankings—it's that small data changes can completely reorder your top features, destroying reproducibility.

What You Should Do

Before trusting feature importance, check feature correlations. Group highly correlated features conceptually, or use techniques like PCA to combine them. Recognize that importance scores for redundant features should be interpreted as a group, not individually.

Key Takeaway: Correlated features split importance scores unpredictably, making individual rankings unreliable; always examine feature relationships before interpreting importance.