Course contentsShow
← Machine Learning and Deep Learning
Lesson 300 of 3,5386. Tree-Based MethodsFree lesson

Random Forests: Adding Feature Randomness

How random forests extend bagging by randomly selecting feature subsets at each split to decorrelate trees.

Random Forests: Adding Feature Randomness

What you'll learn: How random forests build on bagging by introducing an extra layer of randomness that makes tree ensembles even more powerful.

Beyond Pure Bagging

You've learned that bagging creates diverse decision trees by training each on a different bootstrap sample of your data. Random forests take this one step further: at each split in each tree, they only consider a random subset of features rather than all available features.

Why Add Feature Randomness?

When you use regular bagging with decision trees, the trees often still look similar because they all have access to the same features. If one or two features are very strong predictors, almost every tree will use them at the top splits. This means your trees are correlated — they make similar mistakes together.

Think of it like asking a panel of experts for advice, but they've all read the same book. They'll give you similar answers! Random forests force each "expert" (tree) to consider different information, making their predictions more independent.

How It Works

At each node when deciding how to split:

  1. Randomly select m features from all p available features (typically m = √p for classification)
  2. Only consider these m features for the best split
  3. Choose the best split among these candidates using Gini impurity or entropy

This happens at every single split, not just once per tree. Each split gets its own fresh random subset of features.

The Decorrelation Effect

By limiting feature access at each split, random forests decorrelate the trees. Even if some features are dominant predictors, individual trees must find alternative splitting strategies. When you average many decorrelated predictions, you get much better ensemble performance than with correlated trees.

Key Takeaway: Random forests extend bagging by randomly selecting a subset of features at each split, forcing trees to be more different from each other and producing a stronger, more robust ensemble.