Computing the Normal Equation in NumPy
What you'll learn: How to implement the closed-form solution for linear regression using NumPy's matrix operations.
From Theory to Code
You've learned the Normal Equation formula: θ = (XᵀX)⁻¹Xᵀy. Now it's time to translate this mathematical expression into actual NumPy code. This is where linear algebra meets practical implementation.
Step-by-Step Implementation
The computation breaks down into clear operations:
- Transpose X: Use
X.Tornp.transpose(X) - Matrix multiplication XᵀX: Use
X.T @ Xornp.dot(X.T, X) - Compute the inverse: Use
np.linalg.inv() - Multiply by Xᵀy: Chain the operations together
Here's the complete implementation:
import numpy as np
# X is your feature matrix (with bias column), y is your target vector
theta = np.linalg.inv(X.T @ X) @ X.T @ y
That's it! Three operations chained together give you the optimal parameters.
Computational Considerations
Numerical stability: Direct matrix inversion can be unstable when XᵀX is nearly singular (non-invertible). A more robust alternative uses np.linalg.lstsq(), which employs QR decomposition or SVD under the hood:
theta, residuals, rank, s = np.linalg.lstsq(X, y, rcond=None)
Performance: The Normal Equation has time complexity O(n³) due to matrix inversion, where n is the number of features. For very large feature sets (thousands+), iterative methods like gradient descent become more efficient.
Memory: You must compute and store XᵀX, which requires O(n²) memory.
Key Takeaway: The Normal Equation translates directly into NumPy operations (np.linalg.inv(X.T @ X) @ X.T @ y), but for numerical stability, prefer np.linalg.lstsq() in production code.