Course contentsShow
← Computer Science
Lesson 2823 of 2,87263. Classical ML AlgorithmsFree lesson

Normal Equation: Closed-Form Solution

Solving linear regression analytically using matrix operations without iterative optimization.

Normal Equation: Closed-Form Solution

You'll learn how to solve linear regression instantly using matrix algebra instead of iterative gradient descent.

The Core Idea

Until now, you've been using gradient descent to iteratively improve your linear regression model's parameters—taking small steps downhill until you reach the minimum loss. But there's another way: you can solve for the optimal parameters all at once using a mathematical formula called the Normal Equation.

Think of it like this: gradient descent is like finding the bottom of a valley by walking downhill step by step. The Normal Equation is like having a GPS that tells you the exact coordinates of the lowest point instantly.

How It Works

For linear regression, you're trying to minimize Mean Squared Error. The Normal Equation gives you the optimal weights directly:

θ = (XᵀX)⁻¹Xᵀy

Where:

  • X is your feature matrix (all training examples)
  • y is your label vector
  • θ (theta) contains your optimal parameters
  • The superscript ᵀ means transpose
  • The superscript ⁻¹ means matrix inverse

This formula comes from calculus: taking the derivative of the loss function with respect to the parameters, setting it to zero, and solving algebraically. No learning rate needed. No iterations. Just matrix operations.

When to Use It

The Normal Equation works great for small to medium datasets (up to ~10,000 features). Beyond that, computing the matrix inverse becomes computationally expensive (O(n³) complexity), and gradient descent becomes more practical.

Key Takeaway: The Normal Equation provides an exact, one-step solution to linear regression using matrix algebra, eliminating the need for iterative optimization—but only remains practical for moderately-sized feature sets.