Course contentsShow
← Machine Learning and Deep Learning
Lesson 739 of 3,53818. Regularization Techniques for Deep NetworksFree lesson

Implementing Weight Penalties in PyTorch

Master practical implementation of L1, L2, and elastic net regularization using PyTorch optimizers and manual penalties.

Implementing Weight Penalties in PyTorch

You'll learn how to add L1, L2, and elastic net regularization to your PyTorch models using both built-in optimizer features and manual penalty calculations.

The Core Idea

Now that you understand what L1, L2, and elastic net regularization are mathematically, it's time to actually code them. PyTorch gives you two approaches: using the weight_decay parameter in optimizers (for L2) or manually computing penalties and adding them to your loss function (for L1, L2, or elastic net).

Built-in L2 via Optimizer

The simplest way to add L2 regularization is through the optimizer's weight_decay parameter:

optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)

This applies weight decay (L2) automatically during the update step. No changes to your loss computation needed!

Manual Penalty Implementation

For L1 or elastic net, you'll compute the penalty yourself and add it to the loss:

# L1 regularization
l1_lambda = 0.001
l1_penalty = sum(p.abs().sum() for p in model.parameters())
loss = original_loss + l1_lambda * l1_penalty

# L2 regularization (manual)
l2_lambda = 0.01
l2_penalty = sum((p**2).sum() for p in model.parameters())
loss = original_loss + l2_lambda * l2_penalty

# Elastic net (L1 + L2)
loss = original_loss + l1_lambda * l1_penalty + l2_lambda * l2_penalty

The loop for p in model.parameters() iterates through all weight matrices and bias vectors, computing the penalty across all parameters.

Practical Tip

Use optimizer weight_decay for pure L2 (it's more efficient), but use manual penalties when you need L1, need different penalties for different layers, or want elastic net.

Key Takeaway: PyTorch supports L2 regularization natively through weight_decay in optimizers, while L1 and elastic net require manually computing penalties and adding them to your loss before calling backward().