Impurity Measures: Gini Index
What you'll learn: You'll discover how the Gini index provides a faster, simpler alternative to entropy for measuring node impurity in decision trees.
What Is the Gini Index?
The Gini index (or Gini impurity) measures how often a randomly chosen element from a set would be incorrectly classified if it were randomly labeled according to the distribution of labels in that set. Like entropy, it quantifies the "messiness" or impurity of a node, but uses a simpler mathematical formula.
The Formula
For a node with classes, the Gini index is calculated as:
Gini = 1 - Σ(p_i)²
where p_i is the probability of class i in that node.
How It Works
Imagine a bag of colored marbles. If all marbles are the same color, Gini = 0 (perfectly pure). If they're evenly mixed, Gini is higher. The Gini index ranges from 0 (pure) to 0.5 (maximum impurity for binary classification).
Example: A node with 70 "Yes" and 30 "No" samples:
- Gini = 1 - (0.7² + 0.3²) = 1 - (0.49 + 0.09) = 0.42
Gini vs Entropy
Both measure impurity, but:
- Gini is computationally cheaper (no logarithms)
- Entropy penalizes impurity slightly more strongly
- In practice, they usually produce similar trees
- Gini is the default in many implementations because it's faster
Think of them as two thermometers that read the same temperature using different scales—both tell you how "mixed" your data is.
Key Takeaway: The Gini index offers a faster, simpler alternative to entropy for measuring node impurity, using squared probabilities instead of logarithms while producing comparable decision tree splits.