Feature Importance from Decision Trees
What you'll learn: How decision trees calculate which features matter most by measuring how much each feature reduces impurity when making splits.
The Core Idea
When a decision tree makes splits to separate data, some features help clean up the mess more than others. Feature importance measures how much each feature contributes to reducing impurity (whether that's entropy, Gini, or variance) across all the splits where it's used.
Think of it like organizing a messy room: some organizing decisions (like "sort by color") create much more order than others (like "sort by tiny differences in size"). Features that create cleaner separations get higher importance scores.
How It Works
Each time a decision tree uses a feature to split:
- It calculates the impurity reduction = (impurity before split) - (weighted average of impurities after split)
- This reduction is credited to that feature
- After building the full tree, each feature's importance is the total impurity reduction it provided across all splits where it appeared
- These totals are normalized to sum to 1.0 (or 100%)
Mathematical Intuition
If feature f appears in multiple nodes, its importance is:
Importance(f) = Σ (samples at node / total samples) × impurity reduction at that node
Features that split near the root (affecting more samples) and create purer child nodes score highest.
Practical Value
Feature importance tells you which variables your tree "trusts" most for making decisions. This helps with feature selection, understanding your model, and explaining predictions to stakeholders.
Key Takeaway: Feature importance quantifies each feature's total contribution to reducing impurity throughout the tree, giving you a ranked list of which variables matter most for predictions.