Course contentsShow
← Computer Science
Lesson 2856 of 2,87263. Classical ML AlgorithmsFree lesson

Random Forest Feature Importance

Understanding how random forests measure feature importance through mean decrease in impurity across all trees.

Random Forest Feature Importance

You'll learn how random forests measure which features matter most by tracking how much each feature reduces impurity across all trees.

The Core Idea

After training a random forest, you naturally want to know: which features are actually driving predictions? Random forests provide a built-in answer by calculating feature importance scores based on how much each feature reduces impurity (entropy or Gini index) when splitting nodes.

Here's how it works: Every time a decision tree makes a split on a particular feature, that split reduces the impurity of the resulting child nodes. The forest tracks the total impurity reduction caused by each feature across all splits in all trees. Features that consistently produce "cleaner" splits (better separation of classes or tighter regression values) accumulate higher importance scores.

Computing Mean Decrease in Impurity

For each feature:

  1. Track every split that uses that feature across all trees in the forest
  2. Measure the weighted impurity decrease at each split (weighted by the number of samples reaching that node)
  3. Sum these decreases across all trees
  4. Normalize by the total number of trees

Features that appear higher in trees (near the root) and create purer child nodes will score highest. Features never selected for splitting get zero importance.

Why This Matters

This metric is automatic, interpretable, and requires no extra computation beyond training. It helps you identify which features the model relies on, enabling feature selection, debugging unexpected behavior, and explaining predictions to stakeholders.

Key Takeaway: Random forest feature importance quantifies how much each feature reduces impurity on average across all trees, revealing which variables drive the model's decisions without requiring additional testing or holdout data.