Course contentsShow
← Machine Learning and Deep Learning
Lesson 202 of 3,5383. Linear RegressionFree lesson

Computing the Normal Equation in NumPy

Implementing the closed-form solution using matrix operations and understanding computational considerations.

Computing the Normal Equation in NumPy

What you'll learn: How to implement the closed-form solution for linear regression using NumPy's matrix operations.

From Theory to Code

You've learned the Normal Equation formula: θ = (XᵀX)⁻¹Xᵀy. Now it's time to translate this mathematical expression into actual NumPy code. This is where linear algebra meets practical implementation.

Step-by-Step Implementation

The computation breaks down into clear operations:

  1. Transpose X: Use X.T or np.transpose(X)
  2. Matrix multiplication XᵀX: Use X.T @ X or np.dot(X.T, X)
  3. Compute the inverse: Use np.linalg.inv()
  4. Multiply by Xᵀy: Chain the operations together

Here's the complete implementation:

import numpy as np

# X is your feature matrix (with bias column), y is your target vector
theta = np.linalg.inv(X.T @ X) @ X.T @ y

That's it! Three operations chained together give you the optimal parameters.

Computational Considerations

Numerical stability: Direct matrix inversion can be unstable when XᵀX is nearly singular (non-invertible). A more robust alternative uses np.linalg.lstsq(), which employs QR decomposition or SVD under the hood:

theta, residuals, rank, s = np.linalg.lstsq(X, y, rcond=None)

Performance: The Normal Equation has time complexity O(n³) due to matrix inversion, where n is the number of features. For very large feature sets (thousands+), iterative methods like gradient descent become more efficient.

Memory: You must compute and store XᵀX, which requires O(n²) memory.

Key Takeaway: The Normal Equation translates directly into NumPy operations (np.linalg.inv(X.T @ X) @ X.T @ y), but for numerical stability, prefer np.linalg.lstsq() in production code.