Normal Equation: Closed-Form Solution
You'll learn how to solve linear regression instantly using matrix algebra instead of iterative gradient descent.
The Core Idea
Until now, you've been using gradient descent to iteratively improve your linear regression model's parameters—taking small steps downhill until you reach the minimum loss. But there's another way: you can solve for the optimal parameters all at once using a mathematical formula called the Normal Equation.
Think of it like this: gradient descent is like finding the bottom of a valley by walking downhill step by step. The Normal Equation is like having a GPS that tells you the exact coordinates of the lowest point instantly.
How It Works
For linear regression, you're trying to minimize Mean Squared Error. The Normal Equation gives you the optimal weights directly:
θ = (XᵀX)⁻¹Xᵀy
Where:
Xis your feature matrix (all training examples)yis your label vectorθ(theta) contains your optimal parameters- The superscript
ᵀmeans transpose - The superscript
⁻¹means matrix inverse
This formula comes from calculus: taking the derivative of the loss function with respect to the parameters, setting it to zero, and solving algebraically. No learning rate needed. No iterations. Just matrix operations.
When to Use It
The Normal Equation works great for small to medium datasets (up to ~10,000 features). Beyond that, computing the matrix inverse becomes computationally expensive (O(n³) complexity), and gradient descent becomes more practical.
Key Takeaway: The Normal Equation provides an exact, one-step solution to linear regression using matrix algebra, eliminating the need for iterative optimization—but only remains practical for moderately-sized feature sets.