Optimizer Choice and Learning Rates
What you'll learn: How to select the right optimizer and learning rate when fine-tuning large language models to avoid catastrophic forgetting and training instability.
Why AdamW for Fine-Tuning?
When fine-tuning an LLM, you're not starting from scratch—you're refining a model that already knows a lot. AdamW (Adam with decoupled weight decay) has become the gold standard optimizer for this task because it:
- Adapts learning rates per parameter automatically
- Separates weight decay from gradient updates (unlike standard Adam)
- Handles sparse gradients well (common in language modeling)
- Provides momentum-like behavior for smoother convergence
Think of AdamW as having cruise control that adjusts speed differently for each wheel of your car, preventing you from skidding when fine-tuning the already-trained model.
Learning Rates: Much Smaller Than Pretraining
Here's the critical insight: pretraining might use learning rates like 3e-4 or 6e-4, but fine-tuning typically uses 1e-5 to 5e-5—often 10-100× smaller.
Why? Your pretrained model has already learned fundamental language patterns. A high learning rate would violently overwrite this knowledge with task-specific patterns (catastrophic forgetting). A smaller learning rate makes gentle, incremental adjustments.
Analogy: Pretraining is like sculpting clay from a lump. Fine-tuning is like carefully painting details on a finished statue—you need a steady hand, not aggressive chisel strikes.
Typical Configuration
For a 7B model fine-tuning:
- Optimizer:
AdamW - Learning rate:
1e-5to5e-5 - Weight decay:
0.01to0.1 - Often with linear warmup and cosine decay schedule
Key Takeaway: Fine-tuning LLMs requires AdamW with learning rates 10-100× smaller than pretraining to preserve existing knowledge while adapting to new tasks without destabilizing the model.