Choosing Hyperparameters for Adaptive Optimizers
What you'll learn: How to select practical values for beta parameters, epsilon, and learning rates when using Adam-family optimizers.
The Challenge of Configuration
Now that you understand optimizers like Adam, AdamW, and NAdam, you need to know how to configure them. These optimizers have several hyperparameters that control their behavior, and choosing good values is crucial for successful training.
Key Hyperparameters Explained
Learning Rate (α or lr)
This is your most important knob. For Adam-family optimizers:
- Default starting point:
0.001(1e-3) - Common range:
0.0001to0.01 - Rule of thumb: Start with the default, then tune if training stalls or diverges
Think of it like adjusting the gas pedal—Adam adapts per-parameter, but you still control overall speed.
Beta Parameters (β₁ and β₂)
These control momentum and gradient variance estimation:
- β₁ (momentum): Default
0.9works for nearly all cases - β₂ (gradient variance): Default
0.999is robust - When to adjust: Only tune these if you have specific convergence issues (rare)
The betas determine how much history matters—higher values mean longer memory.
Epsilon (ε)
A tiny constant preventing division by zero:
- Default:
1e-8(0.00000001) - Rarely needs tuning: Only adjust if you see numerical instability
- Larger values (like
1e-7) can sometimes help with very small gradients
Practical Starting Recipe
For most deep learning tasks with Adam or AdamW:
learning_rate = 0.001
beta1 = 0.9
beta2 = 0.999
epsilon = 1e-8
Start here, then adjust only the learning rate first if needed.
Key Takeaway: For Adam-family optimizers, keep beta and epsilon at their defaults (β₁=0.9, β₂=0.999, ε=1e-8) and focus your tuning efforts on the learning rate, typically starting at 0.001.