L2 Regularization (Ridge)
What you'll learn: How L2 regularization prevents overfitting by penalizing large weight values through squared terms.
The Core Idea
L2 regularization, also known as Ridge regression, adds a penalty term to your loss function that punishes large weights. Instead of just minimizing prediction error, the model now minimizes error plus the sum of all squared weights multiplied by a regularization parameter λ (lambda).
The modified loss function looks like:
Total Loss = Original Loss + λ × (sum of squared weights)
Think of it like this: imagine you're trying to draw a smooth curve through data points, but you have a budget constraint. Each weight in your model "costs" money proportional to its square. The bigger the weight, the exponentially more expensive it becomes. L2 regularization forces your model to "spend" its weight budget wisely, favoring many small weights over a few large ones.
How It Differs from L1
Unlike L1 regularization (which uses absolute values and can zero out weights entirely), L2 regularization shrinks all weights toward zero but rarely makes them exactly zero. It's like turning down the volume on all your features rather than muting specific ones.
The Effect
L2 regularization produces models that are more stable and generalize better. When weights are smaller, the model is less sensitive to small changes in input features. The λ parameter controls the strength: larger λ means more aggressive shrinking, smaller λ means less regularization.
Key Takeaway: L2 regularization combats overfitting by adding squared weights to the loss function, encouraging the model to distribute importance across features while keeping all coefficients small and stable.