Course contentsShow
← Machine Learning and Deep Learning
Lesson 833 of 3,53819. PyTorch FundamentalsFree lesson

Learning Rate Scheduling

Using PyTorch schedulers like StepLR, ReduceLROnPlateau, and CosineAnnealingLR to adjust learning rates during training.

Learning Rate Scheduling

What You'll Learn

You'll learn how to automatically adjust your model's learning rate during training using PyTorch's built-in schedulers to improve convergence and final performance.

Why Learning Rate Scheduling Matters

Think of training a neural network like descending a mountain in fog. Early on, you can take big steps (high learning rate) to quickly get down the slope. But as you approach the valley floor, smaller, more careful steps (lower learning rate) help you settle into the exact lowest point rather than bouncing around it.

A fixed learning rate often means choosing between fast initial progress and fine-tuned final results. Learning rate schedulers solve this by automatically adjusting the learning rate according to predefined strategies.

Common PyTorch Schedulers

StepLR

Reduces the learning rate by a factor (gamma) every N epochs:

from torch.optim.lr_scheduler import StepLR

scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
# LR multiplied by 0.1 every 10 epochs

ReduceLROnPlateau

Reduces learning rate when a metric (like validation loss) stops improving:

from torch.optim.lr_scheduler import ReduceLROnPlateau

scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=5)
# Cuts LR in half if loss doesn't improve for 5 epochs

CosineAnnealingLR

Smoothly decreases learning rate following a cosine curve:

from torch.optim.lr_scheduler import CosineAnnealingLR

scheduler = CosineAnnealingLR(optimizer, T_max=50)
# Smoothly anneals LR over 50 epochs

Using Schedulers in Training

Call scheduler.step() after each epoch (or after validation for ReduceLROnPlateau):

for epoch in range(num_epochs):
    # Training loop
    train_loss = train_one_epoch(model, train_loader, optimizer)
    
    # Validation
    val_loss = validate(model, val_loader)
    
    # Step the scheduler
    scheduler.step()  # or scheduler.step(val_loss) for ReduceLROnPlateau

Key Takeaway: Learning rate schedulers automate the process of reducing learning rates during training, helping models converge faster initially while achieving better final accuracy through fine-tuning.