Gradient Boosting for Regression
What you'll learn: How gradient boosting builds an ensemble of regression trees by iteratively predicting and correcting residual errors.
The Core Idea
Gradient boosting for regression works like a team of specialists who learn from each other's mistakes. The first tree makes predictions on your data. Instead of stopping there, you calculate how wrong those predictions were (the residuals or errors). Then the second tree doesn't try to predict the original target—it tries to predict those errors. Each subsequent tree focuses on correcting what the previous trees got wrong.
How It Works: The Residual Chase
Here's the step-by-step process:
- Start simple: Make an initial prediction (often just the mean of all target values)
- Calculate residuals: Find the difference between actual values and current predictions
- Fit a tree to residuals: Train a new tree to predict these errors (these are called pseudo-residuals)
- Update predictions: Add the new tree's predictions (scaled by a learning rate) to your running total
- Repeat: Calculate new residuals based on updated predictions, and build another tree
Each tree is relatively shallow (weak learner), but together they form a powerful predictor. The learning rate controls how much each tree contributes—smaller values mean more trees needed but often better generalization.
The Pseudo-Residual Concept
Think of pseudo-residuals as "what we still need to learn." After tree 1, the residuals show where it failed. Tree 2 learns that pattern. After trees 1+2, new residuals reveal remaining gaps. Tree 3 tackles those. This sequential refinement is what makes boosting so effective.
Key Takeaway: Gradient boosting for regression builds an additive model by training each new tree to predict the residuals (errors) of all previous trees combined, gradually reducing prediction error through sequential correction.