L2 Regularization (Weight Decay) Fundamentals
What You'll Learn
You'll learn how L2 regularization penalizes large weights by adding their squared magnitude to the loss function, encouraging simpler models that generalize better.
The Core Problem
Deep networks can develop very large weight values that create overly complex decision boundaries. These models memorize training data instead of learning general patterns—they overfit.
How L2 Regularization Works
L2 regularization adds a penalty term to your loss function based on the squared magnitude of all weights:
Total Loss = Original Loss + λ × (sum of all weights²)
The hyperparameter λ (lambda) controls how much we care about small weights versus fitting the training data perfectly.
Think of It This Way
Imagine you're packing a suitcase. Without regularization, you might cram in huge, bulky items (large weights). L2 regularization is like adding a fee for each item based on its size squared—suddenly, you prefer many small items over a few giant ones. The result? A smoother, more balanced packing strategy.
Why Squared Magnitude?
Squaring weights means:
- Large weights get penalized much more than small ones (10² = 100, but 2² = only 4)
- The penalty is always positive (negative weights get squared too)
- Gradients become proportional to weight magnitude, naturally pushing weights toward zero
The Effect on Training
During backpropagation, this penalty adds an extra term to weight gradients, effectively "decaying" weights toward zero each update. That's why L2 regularization is also called weight decay—it makes weights shrink slightly with every training step, unless the original loss function strongly demands they stay large.
Key Takeaway: L2 regularization adds a squared-weight penalty (λ × Σw²) to the loss function, discouraging large weights and producing smoother, simpler models that generalize better to new data.