Gradient Boosting: Framework and Intuition
What you'll learn: How gradient boosting builds powerful models by iteratively fitting the errors of previous predictions, treating learning as optimization in function space.
The Core Idea
Remember gradient descent from linear regression? You adjusted parameters to minimize loss by following the negative gradient. Gradient boosting applies the same principle, but instead of tweaking parameters, you're adding new models that point in the direction of steepest error reduction.
Think of it this way: You make a prediction with your first simple model. It's not perfect—there are residuals (errors). Instead of adjusting that model's parameters, you train a second model to predict those residuals. Add its predictions to the first model's output, and you've reduced your error. Repeat this process: each new model learns to fix what the previous ensemble got wrong.
Gradient Descent in Function Space
Traditional gradient descent: θ_new = θ_old - learning_rate × gradient
Gradient boosting: F_new(x) = F_old(x) + learning_rate × new_tree(x)
Here, F(x) is your ensemble prediction function. Each new tree is like a "step" in function space, chosen to move your predictions closer to the true targets. The residuals you fit are actually the negative gradient of the loss function with respect to predictions—hence "gradient" boosting.
Why This Works
By sequentially focusing on mistakes, each model specializes in correcting what previous models missed. Unlike bagging (where models train independently), boosting creates a team where each member learns from the team's collective weaknesses. The "learning rate" controls how much each new model contributes, preventing any single tree from dominating.
Key Takeaway: Gradient boosting treats ensemble learning as gradient descent in function space, where each new model is trained on residuals (gradients) to incrementally improve predictions through sequential error correction.