Course contentsShow
← Machine Learning and Deep Learning
Lesson 3188 of 3,53870. Interpretability: Feature-Based MethodsFree lesson

Tree-Based Feature Importance

Learn how decision trees compute importance through impurity reduction and how it aggregates in random forests and gradient boosting.

Tree-Based Feature Importance

You'll learn how decision trees measure feature importance by tracking impurity reduction, and how ensemble methods aggregate these scores.

The Core Idea

Unlike linear models where coefficients directly tell us feature impact, tree-based models measure importance through impurity reduction. When a decision tree splits on a feature, it calculates how much that split reduces uncertainty (impurity) in the target variable. Features that consistently create "cleaner" splits—separating classes or reducing variance—accumulate higher importance scores.

Think of it like organizing a messy closet: features that help you make the best "keep vs donate" decisions most efficiently are the most important organizers.

How It Works

In a single decision tree:

  1. At each node, the algorithm tests potential splits on different features
  2. The split that most reduces impurity (measured by Gini impurity for classification or variance for regression) wins
  3. That feature gets credit proportional to the impurity reduction and the number of samples at that node
  4. Sum all reductions for each feature across all nodes in the tree

Ensemble Aggregation

Random Forests average feature importance across hundreds of trees. Since each tree sees random subsets of features and data, this averaging reduces variance and provides more stable estimates than a single tree.

Gradient Boosting works similarly but with a twist: later trees correct earlier mistakes, so importance scores reflect both direct predictive power and error-correction contributions.

Practical Note

Tree-based importance favors high-cardinality features (many unique values) since they offer more split opportunities. This can mislead—always cross-check with permutation importance for validation.

Key Takeaway: Tree-based feature importance quantifies how much each feature reduces impurity across splits, with ensemble methods providing robust aggregated scores across multiple trees.