Course contentsShow
← Machine Learning and Deep Learning
Lesson 3189 of 3,53870. Interpretability: Feature-Based MethodsFree lesson

Mean Decrease Impurity (MDI)

Understand Gini importance and entropy-based importance in tree models, including their biases toward high-cardinality features.

Mean Decrease Impurity (MDI)

You'll learn how tree-based models compute feature importance through impurity reduction, and why this method favors high-cardinality features.

Core Concept

Mean Decrease Impurity (MDI) measures feature importance by tracking how much each feature reduces node impurity (Gini or entropy) when used for splits across all trees in a forest.

When a decision tree splits on a feature, it aims to reduce impurity—meaning it separates the data into more homogeneous groups. MDI sums these impurity reductions for each feature across all nodes and all trees, then normalizes to produce importance scores.

Gini vs Entropy-Based Importance

Gini impurity measures how often a randomly chosen element would be incorrectly labeled if randomly labeled according to the class distribution at that node. Entropy measures information gain—how much uncertainty is reduced by the split.

Both approaches produce similar importance rankings, though entropy tends to be slightly more computationally expensive. The choice rarely matters in practice for feature importance interpretation.

The High-Cardinality Bias

Here's the critical limitation: MDI is biased toward high-cardinality features—those with many unique values.

Think of it this way: a feature with 100 unique values has more opportunities to split the data than a binary feature. Even if those splits provide minimal predictive value, the accumulated impurity reductions across many nodes artificially inflate the importance score.

This means a user ID column (extremely high cardinality) might appear more important than truly predictive features, simply because it had more splitting opportunities during training.

Key Takeaway: MDI provides fast, built-in feature importance for tree models through impurity reduction tracking, but systematically overestimates the importance of high-cardinality features—use permutation importance for unbiased comparisons.