Course contentsShow
← Machine Learning and Deep Learning
Lesson 716 of 3,53817. Optimization for Deep LearningFree lesson

Polynomial Decay

Polynomial learning rate reduction that reaches zero at a specified final step, enabling precise schedule control.

Polynomial Decay

What you'll learn: How to smoothly reduce your learning rate from start to finish using a polynomial curve that reaches exactly zero at a predefined training step.

The Core Idea

While exponential decay gradually shrinks your learning rate forever, polynomial decay gives you precise control over when training ends. You specify a starting learning rate, a final step number, and a polynomial power—the schedule calculates a smooth curve between them, guaranteeing the learning rate reaches zero (or a specified minimum) exactly when you want it to.

Think of it like planning a road trip where you need to arrive at exactly 6 PM. Polynomial decay is your GPS calculating the perfect speed profile: start fast, gradually slow down, and coast smoothly to a stop right at your destination—not before, not after.

How It Works

The mathematical form is:

lr = initial_lr × (1 - current_step / max_steps)^power
  • initial_lr: Your starting learning rate
  • current_step: How many training steps you've completed
  • max_steps: The total number of steps you plan to train for
  • power: Controls the curve shape (typically 1.0 to 2.0)

When power = 1.0, you get a linear decay—a straight line from start to finish. Higher powers create curves that decay more slowly at first, then drop faster near the end. Lower powers (like 0.5) decay quickly early, then taper off gently.

Why It Matters

Polynomial decay is ideal when you know exactly how long you want to train. It prevents the learning rate from lingering at tiny values (unlike exponential decay) and ensures your optimization completes cleanly at a predetermined endpoint—perfect for reproducible experiments with fixed training budgets.

Key Takeaway: Polynomial decay provides a smooth, controllable learning rate reduction that reaches exactly zero at your specified final training step, giving you precise schedule control over the entire optimization process.