Course contentsShow
← Machine Learning and Deep Learning
Lesson 312 of 3,5386. Tree-Based MethodsFree lesson

Gradient Boosting for Regression

Implementing gradient boosted regression trees and understanding how each tree predicts pseudo-residuals.

Gradient Boosting for Regression

What you'll learn: How gradient boosting builds an ensemble of regression trees by iteratively predicting and correcting residual errors.

The Core Idea

Gradient boosting for regression works like a team of specialists who learn from each other's mistakes. The first tree makes predictions on your data. Instead of stopping there, you calculate how wrong those predictions were (the residuals or errors). Then the second tree doesn't try to predict the original target—it tries to predict those errors. Each subsequent tree focuses on correcting what the previous trees got wrong.

How It Works: The Residual Chase

Here's the step-by-step process:

  1. Start simple: Make an initial prediction (often just the mean of all target values)
  2. Calculate residuals: Find the difference between actual values and current predictions
  3. Fit a tree to residuals: Train a new tree to predict these errors (these are called pseudo-residuals)
  4. Update predictions: Add the new tree's predictions (scaled by a learning rate) to your running total
  5. Repeat: Calculate new residuals based on updated predictions, and build another tree

Each tree is relatively shallow (weak learner), but together they form a powerful predictor. The learning rate controls how much each tree contributes—smaller values mean more trees needed but often better generalization.

The Pseudo-Residual Concept

Think of pseudo-residuals as "what we still need to learn." After tree 1, the residuals show where it failed. Tree 2 learns that pattern. After trees 1+2, new residuals reveal remaining gaps. Tree 3 tackles those. This sequential refinement is what makes boosting so effective.

Key Takeaway: Gradient boosting for regression builds an additive model by training each new tree to predict the residuals (errors) of all previous trees combined, gradually reducing prediction error through sequential correction.