Course contentsShow
← Machine Learning and Deep Learning
Lesson 724 of 3,53817. Optimization for Deep LearningFree lesson

Choosing and Tuning LR Schedules

Guidelines for selecting appropriate schedules based on model, dataset, and training budget, plus hyperparameter tuning strategies.

Choosing and Tuning LR Schedules

What You'll Learn

You'll learn practical guidelines for selecting the right learning rate schedule for your model and how to tune its hyperparameters effectively.

The Art of Schedule Selection

Now that you know what various schedules do, the question becomes: which one should you use? The answer depends on three key factors: your model architecture, dataset characteristics, and training budget.

Model and Dataset Considerations

For quick experiments or small datasets: Start with ReduceLROnPlateau — it's adaptive and requires minimal tuning. Just monitor validation loss and let it decide when to reduce the learning rate.

For large-scale training with known epoch budgets: Cosine annealing or step decay work well. If you have 100 epochs, cosine annealing smoothly reduces the LR from start to finish without manual milestone selection.

For resource-constrained scenarios: One Cycle Policy maximizes performance in limited time by aggressively exploring high learning rates early, then converging quickly.

Hyperparameter Tuning Strategy

Think of schedule tuning like adjusting a recipe:

  1. Start simple: Begin with no schedule — just a fixed LR with your chosen optimizer
  2. Add warmup: If training is unstable early on, add 5-10% of total steps as linear warmup
  3. Choose decay pattern: Based on your training budget, pick step decay (if you know good milestones) or cosine annealing (for smooth reduction)
  4. Tune aggressiveness: Adjust decay factors (step), T_max (cosine), or patience (plateau-based)

Common Combinations

  • Adam + Cosine Annealing: Popular for transformers and vision models
  • SGD + Step Decay: Classic choice for CNNs like ResNet
  • AdamW + One Cycle: Fast convergence for fixed-budget training

Key Takeaway: Choose schedules based on training duration and dataset size; start simple with adaptive schedules, then add complexity only if needed. Always pair warmup with aggressive schedules.