AdamW: Decoupled Weight Decay
What You'll Learn
You'll understand how AdamW fixes a subtle but important flaw in how Adam interacts with weight regularization.
The Problem with Adam and L2 Regularization
When we use L2 regularization (also called weight decay), we want to penalize large weights by adding a term to the loss. However, Adam's adaptive learning rates interact with L2 regularization in an unexpected way.
In traditional SGD, adding L2 regularization to the loss is mathematically equivalent to directly shrinking weights by a small amount at each step. But with Adam, this equivalence breaks down! Because Adam scales gradients differently for each parameter, the L2 penalty gets scaled too, making regularization weaker for parameters with large gradients and stronger for those with small gradients. This isn't what we want.
How AdamW Fixes It
AdamW ("Adam with decoupled Weight decay") separates weight decay from the gradient-based update.
Think of it like this: imagine you're adjusting recipes (parameters) based on taste-test feedback (gradients), while also trying to keep portion sizes reasonable (regularization). The original Adam would adjust your portion control based on how much you changed the recipe—that's weird! AdamW says: "No, let's keep portion control separate from recipe tweaking."
The key change:
- Adam: Adds weight penalty to gradient, then applies adaptive scaling
- AdamW: Applies adaptive gradient update first, then applies weight decay directly to the weights
This simple decoupling makes regularization work as intended and often improves generalization.
When to Use AdamW
If you're using Adam and want regularization benefits, prefer AdamW. It's become the default choice in modern deep learning, especially for large models like transformers.
Key Takeaway: AdamW fixes Adam's weight decay by applying regularization directly to weights instead of mixing it with gradient updates, making regularization more effective and predictable.