Course contentsShow
← Machine Learning and Deep Learning
Lesson 736 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

L1 Regularization for Sparsity

Explore L1 penalty and how it encourages sparse weight solutions by penalizing absolute values of parameters.

L1 Regularization for Sparsity

What you'll learn: How L1 regularization encourages neural networks to set many weights to exactly zero, creating sparse solutions.

The Core Idea

While L2 regularization (which you've just learned) penalizes the squared magnitude of weights, L1 regularization penalizes the absolute value of weights. This seemingly small difference has a profound effect: L1 actively pushes weights toward exactly zero, rather than just making them small.

The L1 penalty adds this term to your loss function:

Loss_total = Loss_original + λ * Σ|w|

Where λ (lambda) controls regularization strength and |w| is the absolute value of each weight.

Why L1 Creates Sparsity

Think of it like cleaning out a closet. L2 regularization says "make everything smaller" — you'd shrink all your belongings proportionally. L1 regularization says "pay a fixed cost for each item you keep" — suddenly it makes sense to throw away (zero out) many items entirely rather than keeping everything tiny.

Mathematically, the gradient of L1 is constant (+1 or -1 depending on sign), not proportional to weight size like L2. This creates consistent pressure to eliminate small weights completely.

The Result: Sparse Networks

Networks trained with L1 regularization end up with many weights at exactly zero. This creates:

  • Feature selection: The network automatically identifies which connections matter
  • Interpretability: Fewer active connections are easier to understand
  • Efficiency: Sparse weights can be stored and computed more efficiently

When to Use L1 vs L2

Use L1 when you suspect many features are irrelevant or want automatic feature selection. Use L2 (weight decay) when you want all weights small but active. Many practitioners combine both (Elastic Net).

Key Takeaway: L1 regularization penalizes the absolute value of weights with a constant gradient, driving many weights to exactly zero and producing sparse, interpretable networks.