Extremely Randomized Trees (Extra Trees)
What you'll learn: How Extra Trees introduce even more randomness than Random Forests by randomizing split thresholds, trading a small increase in bias for significant variance reduction.
The Core Idea
Recall that Random Forests already add randomness by:
- Using bootstrap samples (bagging)
- Randomly selecting a subset of features at each split
Extra Trees (Extremely Randomized Trees) push this randomness further: instead of searching for the optimal split threshold for each feature, Extra Trees randomly select the split threshold for each candidate feature. The algorithm still evaluates these random splits and picks the best one among them, but it never performs an exhaustive search for the globally optimal threshold.
Why Add More Randomness?
Think of it like throwing darts at a board: Random Forests carefully aim each dart within a random section of the board. Extra Trees throw the darts more carelessly within those sections. Each individual throw is less accurate (higher bias), but the overall pattern of many throws becomes more stable and less sensitive to small changes in training data (lower variance).
This extra randomization:
- Reduces correlation between trees even further
- Speeds up training (no threshold optimization needed)
- Reduces overfitting through variance reduction
- Adds a bit more bias, but the variance reduction often wins
Another Key Difference
Extra Trees typically use the full dataset for each tree rather than bootstrap samples, though this is implementation-dependent.
Key Takeaway: Extra Trees add randomness to both feature selection and split thresholds, trading a small bias increase for stronger variance reduction and faster training compared to Random Forests.