Course contentsShow
← Machine Learning and Deep Learning
Lesson 707 of 3,53817. Optimization for Deep LearningFree lesson

AdamW: Decoupled Weight Decay

Learn how AdamW fixes Adam's weight decay implementation by decoupling it from the adaptive learning rate mechanism.

AdamW: Decoupled Weight Decay

What You'll Learn

You'll learn how AdamW fixes a subtle but important flaw in Adam's handling of weight decay, making regularization more effective.

The Problem with Adam's Weight Decay

In standard Adam, weight decay (L2 regularization) is applied by adding a penalty term to the gradient before adaptive learning rates are computed. This means the weight decay gets scaled by Adam's adaptive mechanism, which weakens its regularizing effect—especially for parameters with large gradients that receive smaller effective learning rates.

Think of it like trying to enforce a speed limit (weight decay) but applying it differently depending on how fast someone is already going. The rule becomes inconsistent.

How AdamW Fixes This

AdamW decouples weight decay from the gradient-based update. Instead of mixing weight decay into the gradient calculation, AdamW:

  1. Computes the standard Adam update using momentum and adaptive learning rates
  2. Then separately applies weight decay directly to the weights themselves

Mathematically, where Adam does:

θ ← θ - learning_rate * (m̂ / (√v̂ + ε) + λ * θ)

AdamW does:

θ ← θ - learning_rate * (m̂ / (√v̂ + ε)) - learning_rate * λ * θ

The second term is applied independently, ensuring weight decay has consistent strength regardless of gradient magnitudes.

Why This Matters

By decoupling weight decay, AdamW provides more predictable regularization. The weight decay hyperparameter λ now has a clearer, more consistent effect across all parameters, making your models easier to tune and often improving generalization.

Key Takeaway: AdamW separates weight decay from adaptive learning rates, applying it directly to weights for more effective and consistent regularization compared to standard Adam.