Learning Rate Scheduling
What You'll Learn
You'll learn how to automatically adjust your model's learning rate during training using PyTorch's built-in schedulers to improve convergence and final performance.
Why Learning Rate Scheduling Matters
Think of training a neural network like descending a mountain in fog. Early on, you can take big steps (high learning rate) to quickly get down the slope. But as you approach the valley floor, smaller, more careful steps (lower learning rate) help you settle into the exact lowest point rather than bouncing around it.
A fixed learning rate often means choosing between fast initial progress and fine-tuned final results. Learning rate schedulers solve this by automatically adjusting the learning rate according to predefined strategies.
Common PyTorch Schedulers
StepLR
Reduces the learning rate by a factor (gamma) every N epochs:
from torch.optim.lr_scheduler import StepLR
scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
# LR multiplied by 0.1 every 10 epochs
ReduceLROnPlateau
Reduces learning rate when a metric (like validation loss) stops improving:
from torch.optim.lr_scheduler import ReduceLROnPlateau
scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=5)
# Cuts LR in half if loss doesn't improve for 5 epochs
CosineAnnealingLR
Smoothly decreases learning rate following a cosine curve:
from torch.optim.lr_scheduler import CosineAnnealingLR
scheduler = CosineAnnealingLR(optimizer, T_max=50)
# Smoothly anneals LR over 50 epochs
Using Schedulers in Training
Call scheduler.step() after each epoch (or after validation for ReduceLROnPlateau):
for epoch in range(num_epochs):
# Training loop
train_loss = train_one_epoch(model, train_loader, optimizer)
# Validation
val_loss = validate(model, val_loader)
# Step the scheduler
scheduler.step() # or scheduler.step(val_loss) for ReduceLROnPlateau
Key Takeaway: Learning rate schedulers automate the process of reducing learning rates during training, helping models converge faster initially while achieving better final accuracy through fine-tuning.