Random Forests: Adding Feature Randomness
What you'll learn: How random forests build on bagging by introducing an extra layer of randomness that makes tree ensembles even more powerful.
Beyond Pure Bagging
You've learned that bagging creates diverse decision trees by training each on a different bootstrap sample of your data. Random forests take this one step further: at each split in each tree, they only consider a random subset of features rather than all available features.
Why Add Feature Randomness?
When you use regular bagging with decision trees, the trees often still look similar because they all have access to the same features. If one or two features are very strong predictors, almost every tree will use them at the top splits. This means your trees are correlated — they make similar mistakes together.
Think of it like asking a panel of experts for advice, but they've all read the same book. They'll give you similar answers! Random forests force each "expert" (tree) to consider different information, making their predictions more independent.
How It Works
At each node when deciding how to split:
- Randomly select
mfeatures from allpavailable features (typicallym = √pfor classification) - Only consider these
mfeatures for the best split - Choose the best split among these candidates using Gini impurity or entropy
This happens at every single split, not just once per tree. Each split gets its own fresh random subset of features.
The Decorrelation Effect
By limiting feature access at each split, random forests decorrelate the trees. Even if some features are dominant predictors, individual trees must find alternative splitting strategies. When you average many decorrelated predictions, you get much better ensemble performance than with correlated trees.
Key Takeaway: Random forests extend bagging by randomly selecting a subset of features at each split, forcing trees to be more different from each other and producing a stronger, more robust ensemble.