Course contentsShow
← Machine Learning and Deep Learning
Lesson 311 of 3,5386. Tree-Based MethodsFree lesson

Gradient Boosting: Framework and Intuition

Understanding gradient boosting as gradient descent in function space, fitting residuals iteratively.

Gradient Boosting: Framework and Intuition

What you'll learn: How gradient boosting builds powerful models by iteratively fitting the errors of previous predictions, treating learning as optimization in function space.

The Core Idea

Remember gradient descent from linear regression? You adjusted parameters to minimize loss by following the negative gradient. Gradient boosting applies the same principle, but instead of tweaking parameters, you're adding new models that point in the direction of steepest error reduction.

Think of it this way: You make a prediction with your first simple model. It's not perfect—there are residuals (errors). Instead of adjusting that model's parameters, you train a second model to predict those residuals. Add its predictions to the first model's output, and you've reduced your error. Repeat this process: each new model learns to fix what the previous ensemble got wrong.

Gradient Descent in Function Space

Traditional gradient descent: θ_new = θ_old - learning_rate × gradient

Gradient boosting: F_new(x) = F_old(x) + learning_rate × new_tree(x)

Here, F(x) is your ensemble prediction function. Each new tree is like a "step" in function space, chosen to move your predictions closer to the true targets. The residuals you fit are actually the negative gradient of the loss function with respect to predictions—hence "gradient" boosting.

Why This Works

By sequentially focusing on mistakes, each model specializes in correcting what previous models missed. Unlike bagging (where models train independently), boosting creates a team where each member learns from the team's collective weaknesses. The "learning rate" controls how much each new model contributes, preventing any single tree from dominating.

Key Takeaway: Gradient boosting treats ensemble learning as gradient descent in function space, where each new model is trained on residuals (gradients) to incrementally improve predictions through sequential error correction.