Course contentsShow
← Computer Science
Lesson 2859 of 2,87263. Classical ML AlgorithmsFree lesson

Gradient Boosting Fundamentals

Understanding how gradient boosting fits new models to the residual errors using gradient descent in function space.

Gradient Boosting Fundamentals

What you'll learn: How gradient boosting builds an ensemble by iteratively fitting new models to residual errors using gradient descent in function space.

The Core Idea

While AdaBoost reweights misclassified examples, gradient boosting takes a different approach: it treats ensemble building as an optimization problem in function space. Instead of adjusting data weights, gradient boosting fits each new model to the residual errors (the mistakes) left by the current ensemble.

Think of it like sculpting: your first model creates a rough shape, then each subsequent model "fills in" the errors—the gaps between your predictions and the truth. Each new model learns to predict what the previous ensemble got wrong.

Gradient Descent in Function Space

Here's where it gets elegant: gradient boosting uses gradient descent, but instead of optimizing parameters of a single model, it optimizes the ensemble function itself.

The "gradient" here is the negative gradient of the loss function with respect to the current predictions. For mean squared error, this gradient is simply the residuals (actual - predicted). Each new tree approximates this gradient, effectively taking a "step" toward lower loss.

The process:

  1. Start with an initial prediction (often the mean)
  2. Calculate residuals: errors the current ensemble makes
  3. Fit a new model to predict those residuals
  4. Add this model to the ensemble (often with a learning rate to prevent overfitting)
  5. Repeat until a stopping criterion is met

Unlike bagging (Random Forests), boosting is sequential—each model depends on the previous ones' performance.

Key Takeaway: Gradient boosting builds powerful ensembles by sequentially fitting models to residual errors, using gradient descent to optimize the ensemble as a function that minimizes prediction loss.