Feature Importance from Random Forests
What you'll learn: How Random Forests measure which features matter most for predictions using impurity-based and permutation-based methods.
Why Feature Importance Matters
You've built a Random Forest that predicts house prices using 20 features. But which features actually drive the predictions? Is square footage more important than zip code? Feature importance answers these questions, helping you understand your model and focus on what matters.
Method 1: Mean Decrease in Impurity
This approach (also called "Gini importance") works just like the single-tree feature importance you learned earlier, but averaged across all trees in the forest.
How it works: Each time a feature is used to split a node, we measure how much it reduced impurity (using Gini or entropy). Sum these reductions across all trees and normalize. Features that consistently create "cleaner" splits get higher scores.
Analogy: Imagine 100 chefs each organizing a messy pantry. If shelf size consistently helps them group items neatly across all 100 kitchens, it's an "important" organizing feature.
This method is fast and built into Random Forests automatically, but has a caveat: it can favor high-cardinality features (those with many unique values).
Method 2: Permutation Importance
A more robust alternative: randomly shuffle one feature's values and measure how much prediction accuracy drops. If accuracy tanks, that feature was important.
The logic: Breaking a crucial feature destroys the model's ability to predict well. Breaking an irrelevant feature? No harm done.
This method is model-agnostic (works with any model) and more reliable, but slower since it requires multiple predictions.
Key Takeaway: Random Forests compute feature importance through mean decrease in impurity (fast, built-in) or permutation-based measures (robust, model-agnostic), helping you identify which features truly drive predictions.