Course contentsShow
← Machine Learning and Deep Learning
Lesson 938 of 3,53822. Computer Vision TasksFree lesson

Learning Rate Considerations for Fine-Tuning

Using differential learning rates to update pretrained layers more slowly than new classification heads.

Learning Rate Considerations for Fine-Tuning

What you'll learn: How to use different learning rates for different parts of your network when fine-tuning pretrained models.

Why Different Learning Rates?

When fine-tuning a pretrained model, you're working with two very different types of layers:

  1. Pretrained layers (early feature extractors) — already learned useful patterns from millions of images
  2. New classification head (final layers) — randomly initialized, knows nothing yet

Using the same learning rate for both is like asking an expert chef and a cooking student to learn at the same pace. The chef needs gentle refinements; the student needs bigger steps to catch up.

Differential Learning Rates Explained

Differential learning rates (also called discriminative fine-tuning) means assigning smaller learning rates to earlier pretrained layers and larger rates to newly added layers.

The typical pattern:

  • Bottom layers (closest to input): 0.001 × base_lr or frozen entirely
  • Middle layers: 0.01 × base_lr
  • Top layers (classification head): 1.0 × base_lr (full rate)

This prevents catastrophic forgetting — where aggressive updates destroy the valuable pretrained features your model already knows.

The Analogy

Think of restoring an antique car. The engine and chassis (pretrained features) need careful, minimal adjustments. But the new custom paint job (classification head) can be applied boldly without risk.

Why It Matters

Without differential rates, you risk "breaking" pretrained features with overly aggressive updates, losing the benefits of transfer learning. Your model might actually perform worse than training from scratch.

Key Takeaway: Use smaller learning rates for pretrained layers and larger rates for new classification heads to preserve learned features while adapting to your specific task efficiently.