Course contentsShow
← Computer Science
Lesson 2820 of 2,87263. Classical ML AlgorithmsFree lesson

Computing Gradients for Linear Regression

Deriving partial derivatives of MSE with respect to weights and bias for gradient updates.

Computing Gradients for Linear Regression

You'll learn: How to derive the partial derivatives of Mean Squared Error with respect to weights and bias, enabling gradient descent to update parameters.

Why We Need Gradients

In the previous lesson, you learned gradient descent conceptually—taking steps opposite to the slope to reach a minimum. But how do we find that slope for our linear regression model? We need explicit formulas for the gradient.

The Math Behind the Slope

Recall our linear model: ŷ = wx + b and our loss function MSE: L = (1/n) Σ(y - ŷ)²

To update our parameters, we need:

  • ∂L/∂w (how loss changes with weight)
  • ∂L/∂b (how loss changes with bias)

Computing ∂L/∂w

Using the chain rule:

  1. Take the derivative of the squared error: 2(y - ŷ)
  2. Multiply by the derivative of ŷ with respect to w, which is x
  3. Average over all training examples: ∂L/∂w = -(2/n) Σ(y - ŷ)·x

Computing ∂L/∂b

Similarly:

  1. The derivative of ŷ with respect to b is simply 1
  2. Result: ∂L/∂b = -(2/n) Σ(y - ŷ)

Intuition

Think of the gradient as a compass pointing uphill. For weight w, the gradient tells us: "If you increase w, the loss will change by this much per unit of change." The x term appears because larger input values amplify weight changes.

Key Takeaway: The gradient formulas ∂L/∂w = -(2/n) Σ(y - ŷ)·x and ∂L/∂b = -(2/n) Σ(y - ŷ) give us the exact direction and steepness needed to update our linear regression parameters during gradient descent.