The sqrt(p) and log2(p) Rules
What you'll learn: Why random forests use specific formulas to decide how many features to consider at each split.
The Core Idea
When building each tree in a random forest, we don't consider all features at every split—that would defeat the purpose of adding randomness. But how many features should we consider? Machine learning practitioners have discovered two simple rules of thumb that work remarkably well:
- For classification: Consider sqrt(p) features (the square root of total features)
- For regression: Consider p/3 features (one-third of total features)
Where p is the total number of features in your dataset.
Why These Numbers?
The Balance: These rules strike a sweet spot between diversity and predictive power.
- Too few features → Trees become too random, like guessing blindly
- Too many features → Trees become too similar, losing the "wisdom of crowds" benefit of ensembles
The Intuition: Think of it like a committee making decisions. If everyone considers all the same information, they'll all reach similar conclusions (high correlation between trees). If each person only looks at 1-2 random facts, their conclusions are useless. But if each expert examines a reasonable subset of evidence, you get diverse yet informed opinions.
Why the difference? Classification problems typically have clearer signal in fewer features (hence the smaller sqrt(p)), while regression problems benefit from considering more features to capture subtle numerical relationships (hence the larger p/3).
Example
If you have 100 features:
- Classification: consider 10 features per split (√100 = 10)
- Regression: consider 33 features per split (100/3 ≈ 33)
Key Takeaway: Random forests use sqrt(p) for classification and p/3 for regression as default rules because they balance tree diversity with predictive strength, preventing trees from being too similar or too random.