Elastic Net: Combining L1 and L2
What you'll learn: How elastic net regularization blends L1 and L2 penalties to get the best of both worlds using a mixing parameter.
The Core Idea
So far you've learned that L2 regularization shrinks weights smoothly and keeps all features, while L1 regularization can drive some weights to exactly zero, creating sparse models. But what if you want both behaviors—some feature selection AND smooth weight shrinkage?
Elastic net gives you exactly that by combining both penalties into a single regularization term.
The Mathematical Form
The elastic net loss function looks like this:
Total Loss = Data Loss + λ₁ * Σ|wᵢ| + λ₂ * Σwᵢ²
Or more commonly, using a mixing parameter α between 0 and 1:
Total Loss = Data Loss + λ * [α * Σ|wᵢ| + (1-α) * Σwᵢ²]
Where:
- α = 1: Pure L1 (Lasso)
- α = 0: Pure L2 (Ridge)
- 0 < α < 1: Elastic net blend
Why Use Elastic Net?
Think of it like mixing paint colors. L1 gives you sharp feature selection but can be unstable with correlated features. L2 gives you stability but keeps everything. Elastic net lets you dial in the exact blend you need.
Key benefits:
- Gets sparsity from L1 (some weights become zero)
- Gets stability from L2 (smooth shrinkage)
- Handles correlated features better than pure L1
- More flexible than using either alone
Key Takeaway: Elastic net regularization uses a mixing parameter to combine L1's sparsity-inducing property with L2's smooth shrinkage, giving you flexible control over both feature selection and weight decay.