Course contentsShow
← Computer Science
Lesson 2849 of 2,87263. Classical ML AlgorithmsFree lesson

Overfitting in Decision Trees

Recognizing how unpruned trees can memorize training data, leading to poor generalization on unseen data.

Overfitting in Decision Trees

What you'll learn: How unpruned decision trees can memorize training data instead of learning generalizable patterns, harming performance on new data.

The Problem with Perfect Trees

Decision trees are powerful precisely because they can keep splitting until every training example is perfectly classified. But this strength becomes a weakness: an unpruned tree can grow deep enough to create a unique leaf for nearly every training sample, essentially memorizing the data rather than learning underlying patterns.

Why Trees Memorize

When you let a tree grow without stopping criteria (like maximum depth or minimum samples per leaf), it keeps partitioning the feature space until it achieves zero training error. The tree learns quirks and noise in the training set—like one specific customer who bought a product despite not fitting the usual profile. These aren't real patterns; they're statistical accidents.

Analogy: Imagine studying for an exam by memorizing every practice problem's exact answer instead of understanding the concepts. You'll ace the practice test but fail when the real exam has slightly different questions.

Recognizing the Signs

An overfitted decision tree typically:

  • Has many levels (high depth)
  • Contains leaves with very few samples
  • Achieves near-perfect accuracy on training data
  • Shows significantly worse accuracy on validation/test data

The gap between training and test performance reveals overfitting—the tree captured training-specific noise instead of generalizable decision rules.

Why It Matters

Overfitted trees make unreliable predictions in production because real-world data always differs slightly from training data. The model's complexity exceeds what the data can justify.

Key Takeaway: Unpruned decision trees easily overfit by growing deep enough to memorize training examples, creating complex rules that don't generalize—this is why stopping criteria and pruning are essential.