Course contentsShow
← Machine Learning and Deep Learning
Lesson 1706 of 3,53837. LLM Fine-Tuning: Full and Parameter-EfficientFree lesson

Optimizer Choice and Learning Rates

Selecting appropriate optimizers (AdamW) and much smaller learning rates than pretraining for stable fine-tuning.

Optimizer Choice and Learning Rates

What you'll learn: How to select the right optimizer and learning rate when fine-tuning large language models to avoid catastrophic forgetting and training instability.

Why AdamW for Fine-Tuning?

When fine-tuning an LLM, you're not starting from scratch—you're refining a model that already knows a lot. AdamW (Adam with decoupled weight decay) has become the gold standard optimizer for this task because it:

  • Adapts learning rates per parameter automatically
  • Separates weight decay from gradient updates (unlike standard Adam)
  • Handles sparse gradients well (common in language modeling)
  • Provides momentum-like behavior for smoother convergence

Think of AdamW as having cruise control that adjusts speed differently for each wheel of your car, preventing you from skidding when fine-tuning the already-trained model.

Learning Rates: Much Smaller Than Pretraining

Here's the critical insight: pretraining might use learning rates like 3e-4 or 6e-4, but fine-tuning typically uses 1e-5 to 5e-5—often 10-100× smaller.

Why? Your pretrained model has already learned fundamental language patterns. A high learning rate would violently overwrite this knowledge with task-specific patterns (catastrophic forgetting). A smaller learning rate makes gentle, incremental adjustments.

Analogy: Pretraining is like sculpting clay from a lump. Fine-tuning is like carefully painting details on a finished statue—you need a steady hand, not aggressive chisel strikes.

Typical Configuration

For a 7B model fine-tuning:

  • Optimizer: AdamW
  • Learning rate: 1e-5 to 5e-5
  • Weight decay: 0.01 to 0.1
  • Often with linear warmup and cosine decay schedule

Key Takeaway: Fine-tuning LLMs requires AdamW with learning rates 10-100× smaller than pretraining to preserve existing knowledge while adapting to new tasks without destabilizing the model.