L2 Regularization: Mathematical Derivation and Gradient
What you'll learn: How L2 regularization mathematically modifies the gradients during backpropagation and why it's called "weight decay."
The Mathematical Modification
You already know L2 regularization adds a penalty term to your loss function. Now let's derive exactly how this changes what happens during optimization.
Your original loss for a single example might be:
L(θ) = Loss(ŷ, y)
With L2 regularization, it becomes:
L_regularized(θ) = Loss(ŷ, y) + (λ/2) * ||θ||²
Where λ is your regularization strength and ||θ||² is the sum of squared weights.
The Gradient Derivation
When you compute gradients during backpropagation, the chain rule gives you:
∂L_regularized/∂θ = ∂Loss/∂θ + λ * θ
The first term is your normal gradient from the loss. The second term λ * θ is the regularization gradient — it's simply proportional to the weight itself!
The Weight Decay Interpretation
During a gradient descent update:
θ_new = θ - η * (∂Loss/∂θ + λ * θ)
= θ - η * ∂Loss/∂θ - η * λ * θ
= (1 - η * λ) * θ - η * ∂Loss/∂θ
Notice that (1 - η * λ) factor? That's weight decay — every update multiplies weights by a number slightly less than 1, shrinking them before applying the gradient. It's like erosion wearing down a rock.
Key Takeaway: L2 regularization adds λ * θ to every weight's gradient, which equivalently multiplies each weight by (1 - η * λ) each step — literally decaying weights toward zero while training.