Course contentsShow
← Computer Science
Lesson 2842 of 2,87263. Classical ML AlgorithmsFree lesson

Impurity Measures: Entropy

Learning entropy as a measure of disorder or uncertainty in a set of class labels at a node.

Impurity Measures: Entropy

What You'll Learn

You'll learn how entropy quantifies the disorder or uncertainty in a set of class labels at a decision tree node, helping determine the best splits.

The Core Idea

When building a decision tree, you need to decide how to split data at each node. But how do you know if a split is "good"? Entropy gives you a mathematical answer by measuring how mixed or impure a set of labels is.

Think of entropy like measuring chaos in a room. If everyone in the room is wearing the same color shirt, there's zero chaos (zero entropy)—perfectly organized. If half wear red and half wear blue, there's maximum chaos for two colors. Entropy captures this numerically.

The Formula

For a node with class labels, entropy is calculated as:

Entropy = -Σ(p_i × log₂(p_i))

where p_i is the proportion of samples belonging to class i, and the sum runs over all classes.

Examples

  • Pure node: All 10 samples are class A → p_A = 1.0 → Entropy = 0 (perfectly organized, no uncertainty)
  • Maximum impurity: 5 samples class A, 5 samples class B → p_A = p_B = 0.5 → Entropy ≈ 1.0 (maximum uncertainty for binary classification)
  • Slightly impure: 8 samples class A, 2 samples class B → p_A = 0.8, p_B = 0.2 → Entropy ≈ 0.72 (some uncertainty, but less)

Why It Matters

The splitting process you learned previously uses entropy to evaluate potential splits. Lower entropy after a split means better separation of classes—you've reduced uncertainty about what label samples have.

Key Takeaway: Entropy measures the disorder or unpredictability in class labels at a node, ranging from 0 (pure, single class) to maximum (evenly distributed classes), guiding decision trees toward informative splits.