AdamW: Decoupled Weight Decay
What You'll Learn
You'll learn how AdamW fixes a subtle but important flaw in Adam's handling of weight decay, making regularization more effective.
The Problem with Adam's Weight Decay
In standard Adam, weight decay (L2 regularization) is applied by adding a penalty term to the gradient before adaptive learning rates are computed. This means the weight decay gets scaled by Adam's adaptive mechanism, which weakens its regularizing effect—especially for parameters with large gradients that receive smaller effective learning rates.
Think of it like trying to enforce a speed limit (weight decay) but applying it differently depending on how fast someone is already going. The rule becomes inconsistent.
How AdamW Fixes This
AdamW decouples weight decay from the gradient-based update. Instead of mixing weight decay into the gradient calculation, AdamW:
- Computes the standard Adam update using momentum and adaptive learning rates
- Then separately applies weight decay directly to the weights themselves
Mathematically, where Adam does:
θ ← θ - learning_rate * (m̂ / (√v̂ + ε) + λ * θ)
AdamW does:
θ ← θ - learning_rate * (m̂ / (√v̂ + ε)) - learning_rate * λ * θ
The second term is applied independently, ensuring weight decay has consistent strength regardless of gradient magnitudes.
Why This Matters
By decoupling weight decay, AdamW provides more predictable regularization. The weight decay hyperparameter λ now has a clearer, more consistent effect across all parameters, making your models easier to tune and often improving generalization.
Key Takeaway: AdamW separates weight decay from adaptive learning rates, applying it directly to weights for more effective and consistent regularization compared to standard Adam.