Course contentsShow
← Computer Science
Lesson 2819 of 2,87263. Classical ML AlgorithmsFree lesson

Gradient Descent Basics

Introduction to iterative optimization by following the negative gradient to minimize the loss function.

Gradient Descent Basics

What you'll learn: How to iteratively improve model parameters by following the downhill slope of the loss function.

The Core Idea

Gradient descent is an iterative optimization algorithm that finds the minimum of a loss function by repeatedly taking steps in the direction of steepest descent. Remember from the previous lesson that we want to minimize Mean Squared Error for linear regression? Gradient descent is how we actually do that.

Think of it like hiking down a foggy mountain. You can't see the bottom, but you can feel which direction slopes downward most steeply beneath your feet. By repeatedly taking small steps in the steepest downward direction, you'll eventually reach the valley floor—even without seeing the whole landscape.

How It Works

The gradient is a mathematical term for the slope (or direction of steepest ascent) of your loss function with respect to your model parameters. The negative gradient points in the opposite direction—downhill, toward lower loss.

The algorithm follows these steps:

  1. Start with initial parameter values (often random)
  2. Calculate the gradient of the loss function at your current position
  3. Take a small step in the opposite direction of the gradient (downhill)
  4. Repeat steps 2-3 until the loss stops decreasing significantly

The size of each step is controlled by a learning rate, a small positive number that prevents overshooting the minimum.

Why It Matters

Without gradient descent, we'd need to solve complex equations analytically—or try random guesses. Gradient descent gives us a practical, general-purpose method to optimize parameters for linear regression and countless other machine learning models.

Key Takeaway: Gradient descent minimizes the loss function by iteratively moving parameters in the direction opposite to the gradient, like walking downhill step by step until reaching the lowest point.