Gini Impurity as a Splitting Criterion
What You'll Learn
You'll discover an alternative to entropy for measuring node purity in decision trees, and learn when to use Gini impurity over entropy.
What is Gini Impurity?
Gini impurity measures the probability of incorrectly classifying a randomly chosen element if you labeled it according to the class distribution at a node. Think of it as asking: "If I blindly picked a sample from this node and randomly guessed its class based on the node's proportions, how often would I be wrong?"
For a node with classes, Gini impurity is calculated as:
Gini = 1 - Σ(p_i²)
where p_i is the proportion of samples belonging to class i.
Gini vs Entropy
Both measure impurity, but they differ subtly:
- Entropy (from your last lesson) uses logarithms and measures information content
- Gini impurity uses squared probabilities and is computationally simpler
- Both reach minimum (0) when a node is pure (all one class)
- Both reach maximum when classes are evenly distributed
Analogy: If entropy asks "How surprised am I?", Gini asks "How often would I guess wrong?"
When to Prefer Gini
- Computational efficiency: Gini is faster (no logarithms)
- Practical performance: Both usually produce similar trees
- Default choice: Scikit-learn uses Gini by default for classification trees
- Slightly different penalization: Gini tends to isolate the most frequent class, while entropy creates more balanced splits
Most practitioners stick with Gini unless they have specific reasons to use entropy.
Key Takeaway: Gini impurity provides a faster, simpler alternative to entropy for measuring node purity in decision trees, making it the practical default for classification tasks.