Tree-Based Feature Importance
You'll learn how decision trees measure feature importance by tracking impurity reduction, and how ensemble methods aggregate these scores.
The Core Idea
Unlike linear models where coefficients directly tell us feature impact, tree-based models measure importance through impurity reduction. When a decision tree splits on a feature, it calculates how much that split reduces uncertainty (impurity) in the target variable. Features that consistently create "cleaner" splits—separating classes or reducing variance—accumulate higher importance scores.
Think of it like organizing a messy closet: features that help you make the best "keep vs donate" decisions most efficiently are the most important organizers.
How It Works
In a single decision tree:
- At each node, the algorithm tests potential splits on different features
- The split that most reduces impurity (measured by Gini impurity for classification or variance for regression) wins
- That feature gets credit proportional to the impurity reduction and the number of samples at that node
- Sum all reductions for each feature across all nodes in the tree
Ensemble Aggregation
Random Forests average feature importance across hundreds of trees. Since each tree sees random subsets of features and data, this averaging reduces variance and provides more stable estimates than a single tree.
Gradient Boosting works similarly but with a twist: later trees correct earlier mistakes, so importance scores reflect both direct predictive power and error-correction contributions.
Practical Note
Tree-based importance favors high-cardinality features (many unique values) since they offer more split opportunities. This can mislead—always cross-check with permutation importance for validation.
Key Takeaway: Tree-based feature importance quantifies how much each feature reduces impurity across splits, with ensemble methods providing robust aggregated scores across multiple trees.