Course contentsShow
← Machine Learning and Deep Learning
Lesson 709 of 3,53817. Optimization for Deep LearningFree lesson

AdaMax and AdaBound Variants

Understand variants like AdaMax using infinity norm and AdaBound that transitions from Adam to SGD during training.

AdaMax and AdaBound Variants

What you'll learn: How AdaMax stabilizes Adam using infinity norms and how AdaBound bridges the gap between Adam and SGD during training.

The Core Idea

By now you've seen Adam (lesson 705) and AdamW (707), but researchers keep refining adaptive optimizers. AdaMax and AdaBound are two clever variants that address specific limitations of standard Adam.

AdaMax: Using the Infinity Norm

Adam uses the L2 norm (squared gradients) to adapt learning rates. AdaMax generalizes this to the L∞ norm (infinity norm), which simply tracks the maximum absolute gradient value over time. This creates a more stable, less noisy estimate of gradient scale.

Think of it like tracking the "worst case" gradient instead of averaging all gradients. This makes AdaMax more robust to outlier gradients and can prevent the learning rate from becoming too small too quickly. The update rule is simpler and often more numerically stable than Adam's square root operations.

AdaBound: Transitioning from Adam to SGD

AdaBound takes a different approach: it starts with Adam's adaptive learning rates early in training (when you need fast convergence), then gradually transitions to SGD with momentum as training progresses (when you want better generalization).

It does this by placing dynamic bounds on the learning rates that tighten over time. Early on, bounds are loose (Adam-like behavior). Later, they converge to a fixed learning rate (SGD-like behavior). The analogy: training wheels that gradually lift as you learn to ride.

Why These Matter

  • AdaMax: More stable than Adam in some scenarios, especially with sparse gradients
  • AdaBound: Combines Adam's speed with SGD's generalization, potentially getting "best of both worlds"

Key Takeaway: AdaMax replaces Adam's L2 norm with L∞ for stability, while AdaBound dynamically transitions from adaptive to fixed learning rates, blending fast convergence with strong generalization.