Course contentsShow
← Machine Learning and Deep Learning
Lesson 710 of 3,53817. Optimization for Deep LearningFree lesson

Choosing Hyperparameters for Adaptive Optimizers

Learn practical guidelines for setting beta parameters, epsilon values, and initial learning rates for Adam-family optimizers.

Choosing Hyperparameters for Adaptive Optimizers

What you'll learn: How to select practical values for beta parameters, epsilon, and learning rates when using Adam-family optimizers.

The Challenge of Configuration

Now that you understand optimizers like Adam, AdamW, and NAdam, you need to know how to configure them. These optimizers have several hyperparameters that control their behavior, and choosing good values is crucial for successful training.

Key Hyperparameters Explained

Learning Rate (α or lr)

This is your most important knob. For Adam-family optimizers:

  • Default starting point: 0.001 (1e-3)
  • Common range: 0.0001 to 0.01
  • Rule of thumb: Start with the default, then tune if training stalls or diverges

Think of it like adjusting the gas pedal—Adam adapts per-parameter, but you still control overall speed.

Beta Parameters (β₁ and β₂)

These control momentum and gradient variance estimation:

  • β₁ (momentum): Default 0.9 works for nearly all cases
  • β₂ (gradient variance): Default 0.999 is robust
  • When to adjust: Only tune these if you have specific convergence issues (rare)

The betas determine how much history matters—higher values mean longer memory.

Epsilon (ε)

A tiny constant preventing division by zero:

  • Default: 1e-8 (0.00000001)
  • Rarely needs tuning: Only adjust if you see numerical instability
  • Larger values (like 1e-7) can sometimes help with very small gradients

Practical Starting Recipe

For most deep learning tasks with Adam or AdamW:

learning_rate = 0.001
beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8

Start here, then adjust only the learning rate first if needed.

Key Takeaway: For Adam-family optimizers, keep beta and epsilon at their defaults (β₁=0.9, β₂=0.999, ε=1e-8) and focus your tuning efforts on the learning rate, typically starting at 0.001.