One Cycle Learning Rate Policy
What you'll learn: How to train neural networks faster by cycling the learning rate from low to high and back down over a single training run.
The Core Idea
The One Cycle Learning Rate Policy is a scheduling strategy that uses one cycle of learning rate changes throughout training. Instead of starting high and steadily decaying (like exponential or step decay), you:
- Ramp up the learning rate from a low value to a maximum
- Ramp down back to a very low value (even lower than the start)
This happens over the entire training period—just one complete cycle.
Why This Works
Think of training like hiking up a mountain to find the best campsite (optimal weights). Starting with a low learning rate helps you get oriented. Then increasing the learning rate pushes you to explore more boldly, escaping shallow valleys (local minima). Finally, decreasing the learning rate lets you carefully settle into the best spot you've discovered.
The counterintuitive middle phase—where you raise the learning rate—acts as a form of regularization, preventing overfitting and helping the model generalize better.
The Momentum Connection
One Cycle Policy typically inverts momentum: as learning rate increases, momentum decreases, and vice versa. When taking big steps (high LR), you need less momentum. When taking small steps (low LR), more momentum helps maintain progress.
This can achieve comparable results in fewer epochs—sometimes 10x fewer than traditional training!
Key Takeaway: One Cycle Learning Rate Policy trains networks faster by using a single learning rate cycle (low → high → very low) combined with inverse momentum scheduling, enabling convergence in significantly fewer epochs.