Ridge Regression: Mathematical Formulation
What you'll learn: How to write the mathematical objective function for Ridge Regression and understand what the regularization parameter controls.
The Ridge Objective Function
In ordinary linear regression, we minimize the Mean Squared Error (MSE). Ridge Regression adds a penalty term to discourage large coefficients:
Ridge Objective = MSE + Penalty Term
Mathematically, for predictions ŷ = Xw (where X is your feature matrix and w is the weight vector):
Minimize: (1/n) Σ(y_i - ŷ_i)² + λ Σw_j²
Or in matrix notation:
Minimize: ||y - Xw||² + λ||w||²
The first part is your familiar squared error loss. The second part, λ||w||², is the L2 penalty — it's the sum of the squared coefficients multiplied by lambda.
Understanding Lambda (λ)
Lambda is your regularization parameter that controls the strength of the penalty:
- λ = 0: No penalty → ordinary linear regression
- Small λ: Gentle penalty → coefficients shrink slightly
- Large λ: Strong penalty → coefficients shrink heavily toward zero
Think of λ as a volume knob: turn it up to add more regularization (simpler model), turn it down for less (more complex model).
Why This Works
By penalizing large coefficients, Ridge forces the model to spread influence across features rather than relying heavily on a few. This reduces overfitting because extreme coefficient values usually indicate the model is fitting noise.
Key Takeaway: Ridge Regression minimizes MSE plus λ times the sum of squared coefficients; lambda controls how much we sacrifice fitting the training data to keep coefficients small and the model generalizable.