Course contentsShow
← Mathematics
Lesson 2235 of 3,12146. Linear Algebra — Inner Products and OrthogonalityFree lesson

Normal Equations

Derive and solve the normal equations A^T Ax = A^T b to find the least squares solution.

Normal Equations

You'll learn how to derive and solve the system A^T Ax = A^T b to find the best approximate solution when Ax = b has no exact solution.

Why Normal Equations?

When you're faced with an inconsistent system Ax = b (more equations than unknowns, or incompatible constraints), you can't solve it exactly. But you can find the vector x that minimizes the error ||b - Ax||². This is the least squares solution, and the normal equations give you a direct way to compute it.

Deriving the Normal Equations

Remember from the last lesson: the least squares solution occurs when the error vector (b - Ax) is orthogonal to the column space of A. Mathematically, this means:

A^T(b - Ax) = 0

Expanding this:

  • A^T b - A^T Ax = 0
  • A^T Ax = A^T b

These are the normal equations. The name comes from "normal" meaning perpendicular—the residual is normal to the column space.

Solving the Normal Equations

If A has full column rank (linearly independent columns), then A^T A is invertible, and you can solve:

x = (A^T A)^(-1) A^T b

This gives you the unique least squares solution. Notice how the projection matrix P = A(A^T A)^(-1)A^T appears here—you're essentially projecting b onto the column space of A.

Example Structure

For a simple case where A is 3×2 (three data points, fitting a line), A^T A becomes a 2×2 matrix you can invert by hand, and A^T b is a 2×1 vector. Solve the 2×2 system to get your best-fit parameters.

Key Takeaway: The normal equations A^T Ax = A^T b transform an unsolvable inconsistent system into a solvable one that minimizes squared error, leveraging orthogonality to find the closest possible solution.