Tree-Based Feature Importance
What you'll learn: How to leverage decision trees and ensemble models to automatically rank which features matter most for predictions.
Why Trees Know What Matters
Tree-based models (like decision trees, random forests, and gradient boosting machines) make decisions by splitting data based on feature values. When a feature consistently creates splits that improve prediction accuracy, it's clearly important. When a feature rarely gets used or doesn't improve predictions much, it's less valuable.
Think of it like a detective solving cases: if certain clues (features) repeatedly help crack cases while others are red herrings, you'd naturally trust the useful clues more.
How Feature Importance is Calculated
Tree models track two key metrics:
- How often a feature is used for splitting across all trees
- How much each split improves the model's prediction quality (measured by reduction in impurity like Gini or entropy)
Features that create "good" splits early in trees (closer to the root) and frequently across many trees receive higher importance scores. The final importance is typically normalized to sum to 1.0 or 100%.
Practical Application
After training a random forest or gradient boosting model, you can extract a feature importance array. Higher scores indicate more predictive power. You might then:
- Drop features with near-zero importance to simplify your model
- Focus domain expertise on understanding why certain features dominate
- Use the top N features for faster retraining or deployment
This is an embedded method (from your previous lesson on L1 regularization) because feature selection happens naturally during model training, not as a separate preprocessing step.
Key Takeaway: Tree-based feature importance provides an automatic, interpretable ranking of features based on how much they contribute to splitting decisions and prediction accuracy across the ensemble.