Course contentsShow
← Computer Science
Lesson 2855 of 2,87263. Classical ML AlgorithmsFree lesson

Random Forests: Bagging with Decision Trees

Exploring how random forests combine bagged decision trees with random feature selection at each split for robust predictions.

Random Forests: Bagging with Decision Trees

What you'll learn: How random forests enhance bagging by adding random feature selection to create even more robust ensemble models.

The Core Idea

You've learned that bagging trains multiple models on bootstrap samples of your data and averages their predictions. When you apply bagging specifically to decision trees, you get better results than a single tree—but there's still a problem: the bagged trees tend to look quite similar because they all have access to the same features and will often split on the strongest predictors first.

Random forests solve this by introducing an additional layer of randomness: at each split in each tree, the algorithm only considers a random subset of features rather than all available features. This forces trees to explore different splitting patterns and become more diverse from one another.

Why This Matters

Think of it like asking a panel of experts for advice. If all experts read the same books and focus on the same evidence, their opinions will be similar (correlated). But if each expert specializes in different aspects of the problem, you get more diverse, independent perspectives—and averaging those independent views yields better collective wisdom.

By restricting each split to a random subset of features (commonly √p features for classification or p/3 for regression, where p is total features), random forests decorrelate the trees. This reduces variance even further than standard bagging while maintaining low bias.

The predictions combine multiple independent "voters" rather than multiple similar ones, making the ensemble dramatically more robust to outliers, noise, and overfitting.

Key Takeaway: Random forests extend bagging by randomly limiting feature choices at each split, creating diverse decision trees that collectively produce more accurate and stable predictions than either single trees or standard bagged trees.