Ensemble Learning: The Wisdom of Crowds
What you'll learn: Why combining predictions from multiple models often beats relying on a single model, and how ensembles balance the bias-variance tradeoff.
The Core Idea
Imagine asking one person to estimate the number of jellybeans in a jar versus asking 100 people and averaging their guesses. Surprisingly, the crowd's average is usually more accurate than most individual guesses—even expert ones. This is ensemble learning: combining multiple models to make better predictions than any single model could alone.
Why Ensembles Work
Individual decision trees are prone to overfitting—they memorize training data noise and perform poorly on new data (high variance). A single tree might make idiosyncratic errors based on the specific training samples it saw.
But when you combine multiple trees trained slightly differently:
- Errors cancel out: One tree's mistake might be corrected by another tree's strength
- Averaging reduces variance: Random fluctuations in individual predictions smooth out
- Collective wisdom emerges: The ensemble captures broader patterns while ignoring individual quirks
The Bias-Variance Connection
Remember the bias-variance tradeoff from lesson 222?
- Single complex tree: Low bias (fits training data well), high variance (unstable predictions)
- Ensemble of trees: Maintains low bias while reducing variance through averaging
This is the magic: ensembles let you keep the flexibility of complex models while controlling their instability. Instead of choosing between a simple, stable model and a complex, unstable one, you get both benefits.
Key Takeaway: Ensemble learning combines multiple models to reduce prediction variance and improve generalization, achieving better performance than any single model by letting diverse predictors "vote" on the final answer.