Cosine Annealing with Warm Restarts
You'll learn: How to periodically reset your learning rate to high values using cosine schedules to help your model escape local minima and better explore the loss landscape.
The Core Idea
Remember cosine annealing, where the learning rate smoothly decreases following a cosine curve? Warm Restarts takes this further by periodically "restarting" the schedule—abruptly jumping the learning rate back up to its initial value, then letting it decay again.
Think of it like hiking in foggy mountains. Regular cosine annealing is walking downhill until you stop. With warm restarts, every so often you take a helicopter ride back up the mountain and start descending from a different angle. This helps you discover valleys (minima) you might have missed and potentially find a deeper one.
Why This Works
When training gets stuck in a local minimum or sharp valley, a sudden boost in learning rate acts like an "escape velocity"—giving the optimizer enough energy to jump out and explore other regions. The cosine decay then helps it settle smoothly into a new, potentially better minimum.
The "warm" in warm restarts means the learning rate doesn't start from scratch—it restarts at a predetermined high value, creating a sawtooth pattern when plotted over time.
The Pattern
- Learning rate starts high
- Decays smoothly via cosine curve
- Suddenly jumps back up (restart)
- Decays again, potentially over a longer or shorter period
- Repeat throughout training
This creates multiple opportunities to find better minima and can improve final model performance compared to a single monotonic decay.
Key Takeaway: Cosine Annealing with Warm Restarts periodically resets the learning rate to high values, helping your model escape local minima and explore the loss landscape more thoroughly, often leading to better convergence.