Course contentsShow
← Machine Learning and Deep Learning
Lesson 722 of 3,53817. Optimization for Deep LearningFree lesson

Cyclical Learning Rates

Oscillating learning rate between minimum and maximum bounds to traverse loss landscape saddle points effectively.

Cyclical Learning Rates

What you'll learn: How to oscillate your learning rate between bounds to help your model escape saddle points and train faster.

The Core Idea

Instead of keeping your learning rate fixed or steadily decreasing it, Cyclical Learning Rates (CLR) make it swing back and forth between a minimum and maximum value throughout training. Think of it like a pendulum—your learning rate rises, peaks, falls, and rises again in repeated cycles.

Why does this help? Deep neural network loss landscapes are full of saddle points—flat regions that look like valleys from some directions but hills from others. A small, fixed learning rate can get stuck there for ages. By periodically increasing the learning rate, you give your optimizer enough momentum to jump out of these traps and explore the landscape more effectively.

How It Works

You define two boundaries:

  • Base learning rate (minimum, e.g., 0.0001)
  • Max learning rate (maximum, e.g., 0.001)

During each cycle, the learning rate smoothly increases from the base to the max, then decreases back down. This pattern repeats every few epochs. The triangular pattern is most common, but you can also use cosine or exponential curves.

The magic happens because:

  1. Low rates allow fine-tuning and convergence
  2. High rates help escape poor local minima and saddle points
  3. Oscillation traverses the loss surface more thoroughly than monotonic schedules

Practical Benefits

CLR often trains faster and reaches better solutions than fixed rates. It also reduces the need to carefully tune a single "perfect" learning rate—the range itself becomes more forgiving.

Key Takeaway: Cyclical Learning Rates systematically vary your learning rate between bounds, helping your optimizer escape saddle points and explore the loss landscape more effectively than static schedules.