Lasso vs Ridge: Geometric Intuition
You'll learn why lasso regression often sets coefficients exactly to zero while ridge regression rarely does, using geometric visualization.
The Geometric Picture
Imagine you're trying to minimize your regression error while staying within a "budget" constraint on coefficient sizes. This creates two competing goals:
- The error contours: Elliptical shapes representing equal error levels (like elevation lines on a map)
- The constraint region: A boundary limiting how large your coefficients can be
The magic happens where these two meet—that's your regularized solution.
Why the Shapes Matter
Ridge (L2) constraint region: Forms a circle (or sphere in higher dimensions). This smooth, round boundary means when the error contours touch it, they typically make contact at a point where both coefficients are non-zero—just shrunk.
Lasso (L1) constraint region: Forms a diamond (or diamond-like polytope in higher dimensions) with sharp corners at the axes. When error contours expand outward seeking the constraint boundary, they often hit these pointy corners first—where one or more coefficients equal exactly zero!
The Sparse Solution Insight
Those sharp corners in lasso's diamond are positioned exactly where coordinates become zero (on the axes). Since many error contours will touch the diamond at these corners, lasso naturally produces sparse solutions—models where some features are completely excluded (coefficient = 0).
Ridge's smooth circle has no such corners, so coefficients shrink toward zero but rarely land exactly on it.
Key Takeaway: Lasso's diamond-shaped constraint region has sharp corners at zero-coefficient points, making sparse solutions geometrically likely, while ridge's circular region lacks these corners and keeps all features with small non-zero weights.