Gradient Boosting Fundamentals
What you'll learn: How gradient boosting builds an ensemble by iteratively fitting new models to residual errors using gradient descent in function space.
The Core Idea
While AdaBoost reweights misclassified examples, gradient boosting takes a different approach: it treats ensemble building as an optimization problem in function space. Instead of adjusting data weights, gradient boosting fits each new model to the residual errors (the mistakes) left by the current ensemble.
Think of it like sculpting: your first model creates a rough shape, then each subsequent model "fills in" the errors—the gaps between your predictions and the truth. Each new model learns to predict what the previous ensemble got wrong.
Gradient Descent in Function Space
Here's where it gets elegant: gradient boosting uses gradient descent, but instead of optimizing parameters of a single model, it optimizes the ensemble function itself.
The "gradient" here is the negative gradient of the loss function with respect to the current predictions. For mean squared error, this gradient is simply the residuals (actual - predicted). Each new tree approximates this gradient, effectively taking a "step" toward lower loss.
The process:
- Start with an initial prediction (often the mean)
- Calculate residuals: errors the current ensemble makes
- Fit a new model to predict those residuals
- Add this model to the ensemble (often with a learning rate to prevent overfitting)
- Repeat until a stopping criterion is met
Unlike bagging (Random Forests), boosting is sequential—each model depends on the previous ones' performance.
Key Takeaway: Gradient boosting builds powerful ensembles by sequentially fitting models to residual errors, using gradient descent to optimize the ensemble as a function that minimizes prediction loss.