Implementing Weight Penalties in PyTorch
You'll learn how to add L1, L2, and elastic net regularization to your PyTorch models using both built-in optimizer features and manual penalty calculations.
The Core Idea
Now that you understand what L1, L2, and elastic net regularization are mathematically, it's time to actually code them. PyTorch gives you two approaches: using the weight_decay parameter in optimizers (for L2) or manually computing penalties and adding them to your loss function (for L1, L2, or elastic net).
Built-in L2 via Optimizer
The simplest way to add L2 regularization is through the optimizer's weight_decay parameter:
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)
This applies weight decay (L2) automatically during the update step. No changes to your loss computation needed!
Manual Penalty Implementation
For L1 or elastic net, you'll compute the penalty yourself and add it to the loss:
# L1 regularization
l1_lambda = 0.001
l1_penalty = sum(p.abs().sum() for p in model.parameters())
loss = original_loss + l1_lambda * l1_penalty
# L2 regularization (manual)
l2_lambda = 0.01
l2_penalty = sum((p**2).sum() for p in model.parameters())
loss = original_loss + l2_lambda * l2_penalty
# Elastic net (L1 + L2)
loss = original_loss + l1_lambda * l1_penalty + l2_lambda * l2_penalty
The loop for p in model.parameters() iterates through all weight matrices and bias vectors, computing the penalty across all parameters.
Practical Tip
Use optimizer weight_decay for pure L2 (it's more efficient), but use manual penalties when you need L1, need different penalties for different layers, or want elastic net.
Key Takeaway: PyTorch supports L2 regularization natively through weight_decay in optimizers, while L1 and elastic net require manually computing penalties and adding them to your loss before calling backward().