Exponential Decay
What you'll learn: How to smoothly reduce your learning rate by multiplying it by a constant factor after each epoch, creating a gradual slowdown in training steps.
The Core Idea
Exponential decay shrinks your learning rate by a fixed percentage at regular intervals (typically each epoch). Instead of dropping suddenly like step decay, you multiply your current learning rate by a constant factor (usually between 0.9 and 0.99) to create a smooth, continuous decline.
Think of it like applying compound interest in reverse—your learning rate doesn't drop by a fixed amount, but rather by a fixed proportion of what it currently is. This means large initial learning rates shrink quickly, while already-small rates decrease more gently.
How It Works
The formula is simple:
learning_rate = initial_lr × (decay_rate)^epoch
Where decay_rate is typically 0.95 or 0.96. After epoch 1, your learning rate becomes initial_lr × 0.96. After epoch 2, it's initial_lr × 0.96², and so on.
Why It Helps
Early in training, you want large steps to explore the loss landscape quickly. As you approach a good solution, smaller steps prevent overshooting the minimum. Exponential decay provides this gradual transition naturally—no abrupt jumps that might destabilize training.
Unlike step decay (which creates sudden drops), exponential decay offers smooth adaptation. Your optimizer can continuously adjust to the shrinking learning rate without experiencing jarring changes in behavior.
Typical Setup
You might start with initial_lr = 0.1 and decay_rate = 0.96. After 10 epochs, your learning rate would be approximately 0.1 × 0.96^10 ≈ 0.066. After 50 epochs: 0.1 × 0.96^50 ≈ 0.013.
Key Takeaway: Exponential decay multiplies the learning rate by a constant factor each epoch, creating smooth, continuous reduction that aggressively shrinks early but slows its decline over time.