The Normal Equation Derivation
What you'll learn: How to derive the exact formula for finding the best-fit parameters in linear regression by minimizing squared error using calculus.
The Core Idea
You already know the Normal Equation gives us β = (X^T X)^(-1) X^T y. But why does this formula work? Let's derive it from scratch using calculus.
Remember our goal: minimize the Mean Squared Error (MSE) between predictions and actual values. In matrix form, the sum of squared errors is:
SSE = (y - Xβ)^T (y - Xβ)
Where X is our feature matrix, β are the parameters we're solving for, and y are the true values.
The Derivation Steps
Step 1: Expand the expression SSE = y^T y - y^T Xβ - β^T X^T y + β^T X^T Xβ
Step 2: Notice that y^T Xβ and β^T X^T y are scalars and equal, so: SSE = y^T y - 2β^T X^T y + β^T X^T Xβ
Step 3: Take the derivative with respect to β and set to zero (finding the minimum): ∂SSE/∂β = -2X^T y + 2X^T Xβ = 0
Step 4: Solve for β: X^T Xβ = X^T y β = (X^T X)^(-1) X^T y
This is the closed-form solution — no iteration needed! We analytically found where the error is minimized by setting the gradient to zero.
Why This Works
Think of the derivative as finding where the error "bowl" is flattest. At the bottom of the bowl (minimum error), the slope is zero in all directions. That's exactly where our formula points us.
Key Takeaway: The Normal Equation is derived by taking the gradient of the sum of squared errors with respect to β, setting it to zero, and solving algebraically — giving us the exact optimal parameters in one calculation.