Adam Hyperparameters and Defaults
What You'll Learn
You'll understand the three key hyperparameters in Adam—beta1, beta2, and epsilon—and why their default values work so well across different problems.
The Three Hyperparameters Explained
When you use Adam optimization (which combines momentum and adaptive learning rates), you need to set three hyperparameters:
beta1 (default: 0.9)
This controls the exponential decay rate for the first moment (the momentum term). Think of it like a car's momentum—beta1 = 0.9 means 90% of your previous velocity carries forward, while 10% comes from the current gradient. A higher value (closer to 1.0) gives you smoother, more stable updates but slower response to changes.
beta2 (default: 0.999)
This controls the exponential decay rate for the second moment (the adaptive learning rate part). It tracks the variance of gradients over time. The value 0.999 means we keep 99.9% of past squared gradient information—this creates very stable estimates of gradient scale, preventing wild oscillations in step sizes.
epsilon (default: 1e-8)
This is a tiny numerical stability constant added to the denominator to prevent division by zero. It's like a safety net—you'll never divide by exactly zero, which would cause your training to crash. The value 1e-8 (0.00000001) is small enough not to affect normal updates but large enough to prevent numerical errors.
Why These Defaults Work
The defaults (0.9, 0.999, 1e-8) were chosen through extensive experimentation and work well because:
- They balance responsiveness (beta1) with stability (beta2)
- Beta2 is much closer to 1.0 than beta1, making gradient scale estimates very stable
- Epsilon is negligible for typical gradient scales
Key Takeaway: Adam's default hyperparameters (beta1=0.9, beta2=0.999, epsilon=1e-8) are well-tuned for most deep learning tasks and rarely need adjustment—they balance momentum smoothness with adaptive learning rate stability.