Course contentsShow
← Machine Learning and Deep Learning
Lesson 717 of 3,53817. Optimization for Deep LearningFree lesson

Cosine Annealing

Smoothly decreasing learning rate following a cosine curve from initial to minimum value over training iterations.

Cosine Annealing

What you'll learn: How to smoothly reduce the learning rate along a cosine curve to improve training convergence and final model performance.

The Core Idea

Cosine Annealing is a learning rate schedule that follows the shape of a cosine curve—starting at your initial learning rate and gradually decreasing to a minimum value over a set number of training iterations. Unlike step decay (which drops suddenly) or exponential decay (which drops quickly then slowly), cosine annealing creates a smooth, wave-like decrease.

Why a Cosine Curve?

Think of training like descending a hill toward a valley. Early on, you want large steps (high learning rate) to make fast progress. As you near the bottom, you want tiny, careful steps to settle into the exact lowest point. The cosine curve naturally provides this behavior: it starts with a moderate decline, accelerates in the middle, then gently approaches the minimum—giving your model time to fine-tune as training ends.

Mathematically, at iteration t out of T total iterations:

learning_rate = min_lr + (initial_lr - min_lr) × (1 + cos(π × t / T)) / 2

This formula ensures the learning rate starts at initial_lr when t=0, smoothly decreases following a cosine wave, and reaches min_lr when t=T.

Practical Benefits

  • Smooth transitions: No jarring drops that might disrupt training momentum
  • Better final convergence: The gentle final approach helps find better local minima
  • Single hyperparameter: Just set the total number of iterations (or epochs)
  • Works well with restarts: Can be combined with periodic "warm restarts" (covered later)

Key Takeaway: Cosine annealing provides a smooth, mathematically elegant way to decrease learning rates that often leads to better final model performance than abrupt schedule changes.