Splitting Criteria: Information Gain and Entropy
What You'll Learn
You'll learn how decision trees measure the "messiness" of data at each node and choose the best feature to split on.
The Core Idea
When building a decision tree, at each node we need to decide: which feature should we split on? We want splits that separate our classes as cleanly as possible—turning a mixed bag of labels into purer groups.
Entropy measures how mixed or impure a set of labels is. Think of it like measuring the chaos in a bag of colored marbles:
- A bag with only red marbles has zero entropy (perfectly pure, no chaos)
- A bag with 50% red and 50% blue has high entropy (maximum chaos)
- A bag with 90% red and 10% blue has low entropy (mostly organized)
Mathematically, for a binary classification problem with proportions p₁ and p₂:
Entropy = -p₁ log₂(p₁) - p₂ log₂(p₂)
Information Gain measures how much entropy we reduce by making a particular split. It's the difference between the parent node's entropy and the weighted average entropy of the child nodes after splitting.
Think of it like organizing a messy room: Information Gain tells us which organizing strategy (which feature to split on) reduces the mess the most.
The decision tree algorithm examines all possible features and splits, calculates the information gain for each, and chooses the one with the highest information gain—the split that best reduces chaos and separates the classes.
An Analogy
Imagine sorting mail: you could split by "contains stamp" or "weighs over 50g". Entropy measures how mixed up bill/personal letters are in each pile. Information Gain tells you which sorting method creates the purest piles.
Key Takeaway: Entropy quantifies impurity in a node; information gain measures how much a split reduces that impurity, guiding the tree to choose the most informative feature at each decision point.