Learning Rate and Layer-Wise Decay
What You'll Learn
You'll learn why different layers in a pretrained BERT model should be updated at different speeds during fine-tuning.
The Core Idea
When fine-tuning BERT, not all layers should learn at the same pace. Layer-wise learning rate decay (also called discriminative fine-tuning) applies progressively smaller learning rates to earlier layers and larger rates to later, task-specific layers.
Think of it like renovating a house: the foundation (early BERT layers) already knows general language patterns and needs gentle adjustments, while the new rooftop addition (task-specific classification head) needs major construction work.
Why This Matters
BERT's early layers capture fundamental linguistic features like syntax and word meanings—universal patterns you want to preserve. The final layers and your newly added task head need to adapt significantly to your specific problem (sentiment analysis, question answering, etc.).
Using the same learning rate everywhere risks:
- Catastrophic forgetting: Aggressive updates destroy pretrained knowledge in early layers
- Undertraining: Tiny updates leave your task head undertrained
The Decay Pattern
A typical setup:
- Task-specific head: Learning rate = 2e-5 (baseline)
- Top BERT layers: 2e-5 × 0.95
- Middle layers: 2e-5 × 0.95²
- Bottom layers: 2e-5 × 0.95³ (and so on)
The decay factor (0.95 here) is a hyperparameter you can tune. Lower layers get exponentially smaller learning rates.
In Practice
You configure layer-wise decay when setting up your optimizer, grouping parameters by layer depth and assigning different learning rates to each group. This preserves the rich pretrained representations while allowing task-specific adaptation.
Key Takeaway: Apply smaller learning rates to early BERT layers (preserve general knowledge) and larger rates to task-specific heads (enable specialized learning).