Learning Rate and Scheduler Selection
What you'll learn: How to choose the right learning rate and scheduler to ensure your fine-tuning job converges smoothly without overshooting or stalling.
Why Learning Rate Matters
The learning rate controls how aggressively your model updates its weights during training. Think of it like adjusting the gas pedal while driving: too much and you'll overshoot your destination, too little and you'll never get there.
For LLM fine-tuning, you typically want a much smaller learning rate than training from scratch—often in the range of 1e-5 to 5e-5. Large language models already contain valuable knowledge; you're making gentle adjustments, not rewriting everything.
Understanding Schedulers
A learning rate scheduler adjusts the learning rate during training. Rather than using a fixed rate throughout, schedulers help you:
- Start strong: Begin with a reasonable learning rate to make initial progress
- Finish smooth: Reduce the rate near the end to fine-tune without destabilizing
Common Scheduler Types
Linear scheduler: Gradually decreases the learning rate from initial value to zero over training. Simple and predictable—great for most fine-tuning tasks where you have a clear epoch budget.
Cosine scheduler: Follows a cosine curve, decreasing smoothly but keeping some learning rate longer in the middle phases. Often produces slightly better results because it allows more exploration before the final convergence phase.
Practical Selection
Start conservative: use 2e-5 with a linear scheduler for your first fine-tuning run. Monitor your validation loss—if it's still decreasing steadily near the end, you might benefit from more epochs or a cosine scheduler. If loss spikes or diverges, reduce the learning rate.
Key Takeaway: Choose learning rates 10-100x smaller than pre-training defaults, and use linear or cosine schedulers to gradually reduce the rate for stable convergence without overshooting optimal performance.