Course contentsShow
← Machine Learning and Deep Learning
Lesson 193 of 3,5383. Linear RegressionFree lesson

The Closed-Form Solution (Normal Equation)

Deriving the analytical formula for optimal parameters using calculus and matrix operations.

The Closed-Form Solution (Normal Equation)

What You'll Learn

You'll discover how to calculate the optimal linear regression parameters directly using a mathematical formula, without iterative optimization.

The Direct Path to Optimal Parameters

Remember learning gradient descent as a way to find the best parameters for linear regression? Well, there's another approach: solving the problem directly with algebra and calculus.

When you have a linear model y = Xw (where X is your data matrix and w are your weights), and you want to minimize the Mean Squared Error, you can use calculus to find where the loss function's gradient equals zero. This "zero gradient" condition reveals the optimal parameters.

The Normal Equation Formula

Through matrix calculus, we derive:

w = (X^T X)^(-1) X^T y*

Where:

  • X^T is the transpose of your feature matrix
  • (X^T X)^(-1) is the inverse of X^T X
  • y is your target vector
  • w* is the optimal parameter vector

This is called the Normal Equation or closed-form solution.

Think of It This Way

Imagine you need to find where a valley's lowest point is. Gradient descent is like walking downhill step-by-step until you reach the bottom. The Normal Equation is like having a map that directly shows you the GPS coordinates of the lowest point—you jump straight there!

When to Use It

The Normal Equation works beautifully for small to medium datasets. However, computing (X^T X)^(-1) becomes computationally expensive when you have many features (complexity grows cubically with the number of features).

Key Takeaway: The Normal Equation provides an analytical, one-step solution to linear regression by solving for the point where the gradient of MSE equals zero, eliminating the need for iterative optimization.