Ridge Regression: Closed-Form Solution
What you'll learn: How to compute the optimal ridge regression parameters directly using a mathematical formula, and why we add the identity matrix.
The Analytical Shortcut
Just like ordinary linear regression has the Normal Equation for finding optimal weights in one step, ridge regression also has a closed-form solution. You don't have to use gradient descent — you can solve it directly with matrix algebra.
The Formula
The ridge regression closed-form solution modifies the Normal Equation by adding a penalty term:
β = (XᵀX + λI)⁻¹Xᵀy
Where:
- β is the vector of optimal coefficients
- X is your feature matrix
- y is your target vector
- λ (lambda) is the regularization strength hyperparameter
- I is the identity matrix (same dimensions as XᵀX)
The Role of the Identity Matrix
The identity matrix I is like adding a small value to the diagonal of XᵀX. This serves two crucial purposes:
- Ensures invertibility: Even when features are highly correlated (multicollinearity), adding λI makes the matrix invertible
- Shrinks coefficients: The λI term "pulls" coefficients toward zero, implementing the L2 penalty
Think of I as a regularization dial: when multiplied by λ, it controls how much shrinkage happens. Larger λ means stronger shrinkage.
When λ = 0
Notice that when λ = 0, the formula becomes (XᵀX)⁻¹Xᵀy — exactly the Normal Equation for ordinary linear regression! Ridge regression generalizes the standard solution.
Key Takeaway: Ridge regression has a closed-form solution that adds λI to the Normal Equation, where the identity matrix ensures numerical stability and implements coefficient shrinkage.