Course contentsShow
← Machine Learning and Deep Learning
Lesson 715 of 3,53817. Optimization for Deep LearningFree lesson

Exponential Decay

Continuously decreasing learning rate by a constant multiplicative factor each epoch for smooth, gradual reduction.

Exponential Decay

What you'll learn: How to smoothly reduce your learning rate by multiplying it by a constant factor after each epoch, creating a gradual slowdown in training steps.

The Core Idea

Exponential decay shrinks your learning rate by a fixed percentage at regular intervals (typically each epoch). Instead of dropping suddenly like step decay, you multiply your current learning rate by a constant factor (usually between 0.9 and 0.99) to create a smooth, continuous decline.

Think of it like applying compound interest in reverse—your learning rate doesn't drop by a fixed amount, but rather by a fixed proportion of what it currently is. This means large initial learning rates shrink quickly, while already-small rates decrease more gently.

How It Works

The formula is simple:

learning_rate = initial_lr × (decay_rate)^epoch

Where decay_rate is typically 0.95 or 0.96. After epoch 1, your learning rate becomes initial_lr × 0.96. After epoch 2, it's initial_lr × 0.96², and so on.

Why It Helps

Early in training, you want large steps to explore the loss landscape quickly. As you approach a good solution, smaller steps prevent overshooting the minimum. Exponential decay provides this gradual transition naturally—no abrupt jumps that might destabilize training.

Unlike step decay (which creates sudden drops), exponential decay offers smooth adaptation. Your optimizer can continuously adjust to the shrinking learning rate without experiencing jarring changes in behavior.

Typical Setup

You might start with initial_lr = 0.1 and decay_rate = 0.96. After 10 epochs, your learning rate would be approximately 0.1 × 0.96^10 ≈ 0.066. After 50 epochs: 0.1 × 0.96^50 ≈ 0.013.

Key Takeaway: Exponential decay multiplies the learning rate by a constant factor each epoch, creating smooth, continuous reduction that aggressively shrinks early but slows its decline over time.