Choosing and Tuning LR Schedules
What You'll Learn
You'll learn practical guidelines for selecting the right learning rate schedule for your model and how to tune its hyperparameters effectively.
The Art of Schedule Selection
Now that you know what various schedules do, the question becomes: which one should you use? The answer depends on three key factors: your model architecture, dataset characteristics, and training budget.
Model and Dataset Considerations
For quick experiments or small datasets: Start with ReduceLROnPlateau — it's adaptive and requires minimal tuning. Just monitor validation loss and let it decide when to reduce the learning rate.
For large-scale training with known epoch budgets: Cosine annealing or step decay work well. If you have 100 epochs, cosine annealing smoothly reduces the LR from start to finish without manual milestone selection.
For resource-constrained scenarios: One Cycle Policy maximizes performance in limited time by aggressively exploring high learning rates early, then converging quickly.
Hyperparameter Tuning Strategy
Think of schedule tuning like adjusting a recipe:
- Start simple: Begin with no schedule — just a fixed LR with your chosen optimizer
- Add warmup: If training is unstable early on, add 5-10% of total steps as linear warmup
- Choose decay pattern: Based on your training budget, pick step decay (if you know good milestones) or cosine annealing (for smooth reduction)
- Tune aggressiveness: Adjust decay factors (step), T_max (cosine), or patience (plateau-based)
Common Combinations
- Adam + Cosine Annealing: Popular for transformers and vision models
- SGD + Step Decay: Classic choice for CNNs like ResNet
- AdamW + One Cycle: Fast convergence for fixed-budget training
Key Takeaway: Choose schedules based on training duration and dataset size; start simple with adaptive schedules, then add complexity only if needed. Always pair warmup with aggressive schedules.