Computing Gradients for Linear Regression
You'll learn: How to derive the partial derivatives of Mean Squared Error with respect to weights and bias, enabling gradient descent to update parameters.
Why We Need Gradients
In the previous lesson, you learned gradient descent conceptually—taking steps opposite to the slope to reach a minimum. But how do we find that slope for our linear regression model? We need explicit formulas for the gradient.
The Math Behind the Slope
Recall our linear model: ŷ = wx + b and our loss function MSE: L = (1/n) Σ(y - ŷ)²
To update our parameters, we need:
- ∂L/∂w (how loss changes with weight)
- ∂L/∂b (how loss changes with bias)
Computing ∂L/∂w
Using the chain rule:
- Take the derivative of the squared error:
2(y - ŷ) - Multiply by the derivative of
ŷwith respect tow, which isx - Average over all training examples: ∂L/∂w = -(2/n) Σ(y - ŷ)·x
Computing ∂L/∂b
Similarly:
- The derivative of
ŷwith respect tobis simply1 - Result: ∂L/∂b = -(2/n) Σ(y - ŷ)
Intuition
Think of the gradient as a compass pointing uphill. For weight w, the gradient tells us: "If you increase w, the loss will change by this much per unit of change." The x term appears because larger input values amplify weight changes.
Key Takeaway: The gradient formulas ∂L/∂w = -(2/n) Σ(y - ŷ)·x and ∂L/∂b = -(2/n) Σ(y - ŷ) give us the exact direction and steepness needed to update our linear regression parameters during gradient descent.