Course contentsShow
← Machine Learning and Deep Learning
Lesson 447 of 3,53810. Feature EngineeringFree lesson

Tree-Based Feature Importance

Extract feature importance scores from tree models to rank and select the most predictive features.

Tree-Based Feature Importance

What you'll learn: How to leverage decision trees and ensemble models to automatically rank which features matter most for predictions.

Why Trees Know What Matters

Tree-based models (like decision trees, random forests, and gradient boosting machines) make decisions by splitting data based on feature values. When a feature consistently creates splits that improve prediction accuracy, it's clearly important. When a feature rarely gets used or doesn't improve predictions much, it's less valuable.

Think of it like a detective solving cases: if certain clues (features) repeatedly help crack cases while others are red herrings, you'd naturally trust the useful clues more.

How Feature Importance is Calculated

Tree models track two key metrics:

  1. How often a feature is used for splitting across all trees
  2. How much each split improves the model's prediction quality (measured by reduction in impurity like Gini or entropy)

Features that create "good" splits early in trees (closer to the root) and frequently across many trees receive higher importance scores. The final importance is typically normalized to sum to 1.0 or 100%.

Practical Application

After training a random forest or gradient boosting model, you can extract a feature importance array. Higher scores indicate more predictive power. You might then:

  • Drop features with near-zero importance to simplify your model
  • Focus domain expertise on understanding why certain features dominate
  • Use the top N features for faster retraining or deployment

This is an embedded method (from your previous lesson on L1 regularization) because feature selection happens naturally during model training, not as a separate preprocessing step.

Key Takeaway: Tree-based feature importance provides an automatic, interpretable ranking of features based on how much they contribute to splitting decisions and prediction accuracy across the ensemble.