Course contentsShow
← Machine Learning and Deep Learning
Lesson 720 of 3,53817. Optimization for Deep LearningFree lesson

ReduceLROnPlateau: Adaptive Scheduling

Monitoring validation metrics and reducing learning rate when performance plateaus, adapting to training dynamics automatically.

ReduceLROnPlateau: Adaptive Scheduling

What You'll Learn

You'll learn how to automatically reduce the learning rate when your model stops improving, letting the training process adapt to performance plateaus without manual intervention.

The Core Idea

Unlike the fixed schedules you've seen (step decay, cosine annealing, etc.), ReduceLROnPlateau monitors a validation metric (like validation loss or accuracy) and reduces the learning rate only when that metric stops improving for a specified number of epochs. Think of it as an autopilot that "feels" when training is stuck and adjusts course automatically.

Why This Matters

Imagine you're descending a mountain in fog. Fixed schedules are like descending at predetermined speeds regardless of terrain. ReduceLROnPlateau is like slowing down when you notice you're not making progress anymore — it responds to what's actually happening rather than following a preset plan.

How It Works

The scheduler tracks your chosen metric (usually validation loss) over time:

  1. Monitor: After each epoch, check the validation metric
  2. Patience: If the metric doesn't improve for patience epochs, reduce the learning rate
  3. Factor: Multiply learning rate by a factor (e.g., 0.1 for 10× reduction)
  4. Repeat: Continue monitoring with the new learning rate

Key Parameters

  • patience: How many epochs to wait before reducing (e.g., 5-10)
  • factor: How much to reduce by (typically 0.1 to 0.5)
  • mode: Whether you're minimizing (loss) or maximizing (accuracy)
  • min_lr: Don't reduce below this threshold

Key Takeaway: ReduceLROnPlateau automatically adapts your learning rate based on validation performance plateaus, making it a reactive, metric-driven scheduling strategy that responds to your model's actual training dynamics rather than a predetermined timeline.