ReduceLROnPlateau: Adaptive Scheduling
What You'll Learn
You'll learn how to automatically reduce the learning rate when your model stops improving, letting the training process adapt to performance plateaus without manual intervention.
The Core Idea
Unlike the fixed schedules you've seen (step decay, cosine annealing, etc.), ReduceLROnPlateau monitors a validation metric (like validation loss or accuracy) and reduces the learning rate only when that metric stops improving for a specified number of epochs. Think of it as an autopilot that "feels" when training is stuck and adjusts course automatically.
Why This Matters
Imagine you're descending a mountain in fog. Fixed schedules are like descending at predetermined speeds regardless of terrain. ReduceLROnPlateau is like slowing down when you notice you're not making progress anymore — it responds to what's actually happening rather than following a preset plan.
How It Works
The scheduler tracks your chosen metric (usually validation loss) over time:
- Monitor: After each epoch, check the validation metric
- Patience: If the metric doesn't improve for
patienceepochs, reduce the learning rate - Factor: Multiply learning rate by a factor (e.g., 0.1 for 10× reduction)
- Repeat: Continue monitoring with the new learning rate
Key Parameters
patience: How many epochs to wait before reducing (e.g., 5-10)factor: How much to reduce by (typically 0.1 to 0.5)mode: Whether you're minimizing (loss) or maximizing (accuracy)min_lr: Don't reduce below this threshold
Key Takeaway: ReduceLROnPlateau automatically adapts your learning rate based on validation performance plateaus, making it a reactive, metric-driven scheduling strategy that responds to your model's actual training dynamics rather than a predetermined timeline.