Course contentsShow
← Machine Learning and Deep Learning
Lesson 735 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

L2 Regularization: Mathematical Derivation and Gradient

Derive how L2 regularization modifies gradients and understand the weight decay interpretation in optimization.

L2 Regularization: Mathematical Derivation and Gradient

What you'll learn: How L2 regularization mathematically modifies the gradients during backpropagation and why it's called "weight decay."

The Mathematical Modification

You already know L2 regularization adds a penalty term to your loss function. Now let's derive exactly how this changes what happens during optimization.

Your original loss for a single example might be:

L(θ) = Loss(ŷ, y)

With L2 regularization, it becomes:

L_regularized(θ) = Loss(ŷ, y) + (λ/2) * ||θ||²

Where λ is your regularization strength and ||θ||² is the sum of squared weights.

The Gradient Derivation

When you compute gradients during backpropagation, the chain rule gives you:

∂L_regularized/∂θ = ∂Loss/∂θ + λ * θ

The first term is your normal gradient from the loss. The second term λ * θ is the regularization gradient — it's simply proportional to the weight itself!

The Weight Decay Interpretation

During a gradient descent update:

θ_new = θ - η * (∂Loss/∂θ + λ * θ)
      = θ - η * ∂Loss/∂θ - η * λ * θ
      = (1 - η * λ) * θ - η * ∂Loss/∂θ

Notice that (1 - η * λ) factor? That's weight decay — every update multiplies weights by a number slightly less than 1, shrinking them before applying the gradient. It's like erosion wearing down a rock.

Key Takeaway: L2 regularization adds λ * θ to every weight's gradient, which equivalently multiplies each weight by (1 - η * λ) each step — literally decaying weights toward zero while training.