Course contentsShow
← Machine Learning and Deep Learning
Lesson 737 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

L1 vs L2: Geometric Interpretation and Trade-offs

Compare L1 and L2 through constraint geometry, understanding when each penalty type is most appropriate.

L1 vs L2: Geometric Interpretation and Trade-offs

What you'll learn: How the geometric shapes of L1 and L2 constraints explain why L1 produces sparse solutions while L2 smoothly shrinks all weights.

The Geometric Picture

Imagine you're searching for optimal weights, but regularization adds a constraint: your solution must stay within a certain region. L2 regularization creates a circular (or spherical) boundary around the origin, while L1 regularization creates a diamond-shaped (or cross-polytope) boundary with sharp corners at the axes.

When the optimizer searches for the best weights that minimize your loss and respect the regularization constraint, it looks for where the loss function's contours first touch the constraint region.

Why L1 Produces Sparsity

The diamond shape of L1 has pointy corners exactly on the axes (where one weight is zero and others are non-zero). Loss function contours are far more likely to first touch these sharp corners than the smooth edges. When they do, one or more weights become exactly zero — creating sparsity.

L2's smooth circular boundary has no special points. The loss contours typically touch somewhere along the smooth curve, shrinking all weights proportionally but rarely driving any to exactly zero.

When to Choose Each

  • Use L1 when you suspect many features are irrelevant and want automatic feature selection. The sparsity helps interpretability and can reduce model size.
  • Use L2 when you believe most features contribute something and want stable, smooth weight shrinkage. L2 is more computationally efficient and numerically stable.
  • Elastic Net combines both: λ₁|w| + λ₂w², capturing benefits of both penalties.

Key Takeaway: L1's sharp geometric constraints produce sparse solutions by pushing weights to exactly zero, while L2's smooth constraints shrink all weights proportionally — choose based on whether you need feature selection or stable regularization.