Course contentsShow
← Machine Learning and Deep Learning
Lesson 212 of 3,5383. Linear RegressionFree lesson

Computing the Gradient of MSE

Derive the partial derivatives of mean squared error with respect to weights and bias analytically.

Computing the Gradient of MSE

You'll learn: How to mathematically derive the partial derivatives of mean squared error with respect to your model's weights and bias.

Why We Need the Gradient

You've learned that the gradient points in the direction of steepest ascent on the cost function landscape. To actually use gradient descent, we need the explicit formula for these derivatives. Think of it like needing turn-by-turn directions rather than just knowing "go downhill."

The Analytical Derivation

Recall that for multiple linear regression, our prediction is:

ŷ = Xw + b

And our MSE cost function is:

J(w, b) = (1/n) Σ(ŷᵢ - yᵢ)²

To find how to update w and b, we need:

  • ∂J/∂w: How does cost change with each weight?
  • ∂J/∂b: How does cost change with bias?

Using the chain rule from calculus:

∂J/∂w = (2/n) Xᵀ(Xw + b - y)

∂J/∂b = (2/n) Σ(ŷᵢ - yᵢ)

These formulas tell us exactly how much each parameter contributes to our error. The term (Xw + b - y) is just our residuals—the prediction mistakes. The gradient essentially says: "weight your input features by how wrong you were."

What This Means Practically

The gradient for weights is a vector the same size as w, telling us how to adjust each feature's coefficient. The gradient for bias is a single number. Both point toward increasing error, which is why we'll subtract them during gradient descent.

Key Takeaway: The gradient of MSE provides explicit formulas—involving residuals and input features—that quantify exactly how each parameter affects the loss, enabling iterative optimization.