Splitting Criteria for Regression Trees
What you'll learn: How regression trees decide where to split data by minimizing prediction error instead of measuring impurity.
The Core Difference from Classification
In classification trees, you've learned to use entropy and Gini index to measure how "mixed" the classes are at a node. Regression trees face a different challenge: predicting continuous numerical values, not categories. So instead of measuring class impurity, we measure prediction error.
Variance Reduction
The most common splitting criterion for regression trees is variance reduction. Here's the intuition:
- At each node, all samples share the same predicted value (typically their mean)
- High variance means the predictions are far from the actual values — that's bad
- A good split creates child nodes where samples are more similar to each other
- Lower variance in children = better predictions
The algorithm:
- Calculate the variance of the target values in the current node
- For each potential split, compute the weighted average variance of the resulting child nodes
- Choose the split that produces the largest variance reduction
Mean Squared Error Alternative
Some implementations use mean squared error (MSE) directly instead of variance — these are equivalent! MSE measures how far predictions are from actual values. Minimizing MSE is the same as maximizing variance reduction.
Analogy: Imagine grouping people by height to predict their weight. A bad split mixes very short and very tall people together. A good split separates them so each group has similar heights — and thus more predictable weights.
Key Takeaway: Regression trees split by finding the threshold that maximizes variance reduction (or minimizes MSE), creating child nodes where the continuous target values are most homogeneous.