Course contentsShow
← Computer Science
Lesson 2843 of 2,87263. Classical ML AlgorithmsFree lesson

Information Gain

Computing information gain to quantify the reduction in entropy achieved by splitting on a particular feature.

Information Gain

You'll learn: How to measure the value of splitting on a feature by computing the reduction in entropy it provides.

What is Information Gain?

Information gain is the metric that answers: "How much uncertainty does this split remove?" After learning about entropy as a measure of impurity, information gain tells us how much purer our subsets become when we choose a particular feature to split on.

Think of it like sorting a messy drawer. If you split items by "sharp vs. dull," you've gained information if each pile is now more organized (less mixed) than the original jumble. If both piles are still equally messy, you've gained nothing.

Computing Information Gain

Information gain is calculated as:

Information Gain = Entropy(parent) - Weighted Average of Entropy(children)

Here's the step-by-step process:

  1. Calculate parent entropy: Measure the impurity of the dataset before splitting
  2. Split on a feature: Divide data into subsets based on feature values
  3. Calculate child entropies: Measure impurity in each resulting subset
  4. Weight by size: Multiply each child's entropy by the fraction of samples it contains
  5. Compute the difference: Subtract the weighted average from the parent entropy

The feature with the highest information gain is the best choice for splitting, because it reduces uncertainty the most.

Example Intuition

Imagine splitting weather data to predict "play tennis?":

  • If splitting by "Outlook" creates very pure subsets (all "yes" or all "no" in each group), information gain is high
  • If splitting by "Temperature" still leaves mixed groups, information gain is low

Key Takeaway: Information gain quantifies how much a split reduces entropy, helping decision trees choose the most informative features at each step.