Course contentsShow
← Machine Learning and Deep Learning
Lesson 297 of 3,5386. Tree-Based MethodsFree lesson

Ensemble Learning: The Wisdom of Crowds

Why combining multiple models often outperforms a single model, and the variance-bias tradeoff in ensembles.

Ensemble Learning: The Wisdom of Crowds

What you'll learn: Why combining predictions from multiple models often beats relying on a single model, and how ensembles balance the bias-variance tradeoff.

The Core Idea

Imagine asking one person to estimate the number of jellybeans in a jar versus asking 100 people and averaging their guesses. Surprisingly, the crowd's average is usually more accurate than most individual guesses—even expert ones. This is ensemble learning: combining multiple models to make better predictions than any single model could alone.

Why Ensembles Work

Individual decision trees are prone to overfitting—they memorize training data noise and perform poorly on new data (high variance). A single tree might make idiosyncratic errors based on the specific training samples it saw.

But when you combine multiple trees trained slightly differently:

  • Errors cancel out: One tree's mistake might be corrected by another tree's strength
  • Averaging reduces variance: Random fluctuations in individual predictions smooth out
  • Collective wisdom emerges: The ensemble captures broader patterns while ignoring individual quirks

The Bias-Variance Connection

Remember the bias-variance tradeoff from lesson 222?

  • Single complex tree: Low bias (fits training data well), high variance (unstable predictions)
  • Ensemble of trees: Maintains low bias while reducing variance through averaging

This is the magic: ensembles let you keep the flexibility of complex models while controlling their instability. Instead of choosing between a simple, stable model and a complex, unstable one, you get both benefits.

Key Takeaway: Ensemble learning combines multiple models to reduce prediction variance and improve generalization, achieving better performance than any single model by letting diverse predictors "vote" on the final answer.