Regression Trees and Variance Reduction
What you'll learn: How decision trees adapt from classification to regression by using variance reduction as their splitting criterion.
From Categories to Continuous Values
You've seen decision trees split data for classification using Gini impurity or entropy. But what happens when you need to predict a continuous value like house price or temperature instead of a category?
Regression trees work the same way structurally—they still create splits to divide your data—but they optimize for a different goal: minimizing variance in the predicted values.
How Variance Reduction Works
At each potential split, a regression tree calculates:
- The variance (spread) of target values in the current node
- The weighted average variance of the two child nodes after splitting
- The variance reduction = parent variance - weighted child variance
The split that achieves the largest variance reduction wins. This is analogous to maximizing information gain in classification, but for continuous targets.
Think of it like organizing books by price: you want each shelf to contain books with similar prices (low variance), making predictions easier.
Making Predictions
Once built, each leaf node contains training samples. The prediction for any new data point is simply the mean (average) of all target values in its leaf. This mean minimizes the mean squared error for that region—connecting back to your MSE loss function knowledge.
The splitting process essentially partitions your feature space into regions where the target variable behaves similarly, reducing overall prediction uncertainty.
Key Takeaway: Regression trees use variance reduction (or equivalently, MSE reduction) as their splitting criterion, predicting the mean target value within each leaf node to handle continuous outputs.