Computing the Gradient of MSE
You'll learn: How to mathematically derive the partial derivatives of mean squared error with respect to your model's weights and bias.
Why We Need the Gradient
You've learned that the gradient points in the direction of steepest ascent on the cost function landscape. To actually use gradient descent, we need the explicit formula for these derivatives. Think of it like needing turn-by-turn directions rather than just knowing "go downhill."
The Analytical Derivation
Recall that for multiple linear regression, our prediction is:
ŷ = Xw + b
And our MSE cost function is:
J(w, b) = (1/n) Σ(ŷᵢ - yᵢ)²
To find how to update w and b, we need:
- ∂J/∂w: How does cost change with each weight?
- ∂J/∂b: How does cost change with bias?
Using the chain rule from calculus:
∂J/∂w = (2/n) Xᵀ(Xw + b - y)
∂J/∂b = (2/n) Σ(ŷᵢ - yᵢ)
These formulas tell us exactly how much each parameter contributes to our error. The term (Xw + b - y) is just our residuals—the prediction mistakes. The gradient essentially says: "weight your input features by how wrong you were."
What This Means Practically
The gradient for weights is a vector the same size as w, telling us how to adjust each feature's coefficient. The gradient for bias is a single number. Both point toward increasing error, which is why we'll subtract them during gradient descent.
Key Takeaway: The gradient of MSE provides explicit formulas—involving residuals and input features—that quantify exactly how each parameter affects the loss, enabling iterative optimization.