Course contentsShow
← Machine Learning and Deep Learning
Lesson 213 of 3,5383. Linear RegressionFree lesson

The Gradient Descent Update Rule

Learn the iterative parameter update formula and understand how learning rate controls step size.

The Gradient Descent Update Rule

What you'll learn: How to update model parameters iteratively using gradients and learning rates to minimize the cost function.

The Core Idea

Now that you understand gradients point toward steepest ascent, gradient descent works by moving in the opposite direction. The update rule is the formula that tells you exactly how to adjust your parameters after each step.

For linear regression with parameters (weights and bias), the update rule looks like:

θ_new = θ_old - α × gradient

Where:

  • θ (theta) represents your parameters (coefficients)
  • α (alpha) is the learning rate — a small positive number you choose
  • gradient is the slope you computed from the MSE cost function

Understanding the Learning Rate

Think of the learning rate as the size of your steps when walking downhill. Too large, and you might overshoot the valley and bounce around wildly. Too small, and you'll take forever to reach the bottom (or get stuck on a plateau).

Typical learning rates range from 0.001 to 0.1, but the right value depends on your specific problem.

The Update Process

  1. Start with initial parameter values (often random or zeros)
  2. Compute the gradient at your current position
  3. Take a step in the opposite direction (subtract the gradient)
  4. The learning rate controls how big that step is
  5. Repeat until convergence

Key Takeaway: The gradient descent update rule θ = θ - α × gradient is the engine of iterative optimization — it moves parameters downhill toward lower cost, with learning rate controlling step size.