Learning Rate Scaling Rules
What you'll learn: How to adjust learning rates when scaling up batch sizes and worker counts in distributed training to maintain training stability and convergence speed.
The Core Problem
When you move from training on a single GPU with batch size 32 to distributed training across 8 GPUs with effective batch size 256, should you keep the same learning rate? The answer is no—and the adjustment strategy matters enormously.
Why Scaling Matters
Larger batch sizes mean fewer gradient updates per epoch. If you naively keep the learning rate constant, your model takes much longer to converge. Think of it like crossing a field: with smaller steps (small batches), you check your direction frequently; with larger strides (large batches), you need to step more boldly or you'll barely move.
Linear Scaling Rule
The most common approach: multiply your learning rate by the same factor you scaled your batch size.
- Original: batch size 32, learning rate 0.1
- Scaled: batch size 256 (8× larger)
- New learning rate: 0.1 × 8 = 0.8
This works well when batch sizes are reasonably large and the optimization landscape is relatively smooth. It's based on the idea that gradient noise decreases with larger batches, so you can afford more aggressive steps.
Square Root Scaling Rule
A more conservative alternative: scale the learning rate by the square root of the batch size increase.
- Scaled: batch size 256 (8× larger)
- New learning rate: 0.1 × √8 ≈ 0.283
This rule is safer when training is unstable or when batch sizes get very large, as it reduces the risk of divergence from overly aggressive learning rates.
Choosing Between Them
Linear scaling is standard for computer vision tasks. Square root scaling helps with noisy gradients, very large batches (>8K), or sensitive models like transformers in early training phases.
Key Takeaway: When increasing batch size by factor k, scale learning rate by k (linear rule) for aggressive training or √k (sqrt rule) for stability—always validate with a learning rate warmup period.