Learning Rate Scaling with Gradient Accumulation
What you'll learn: How to adjust your learning rate when using gradient accumulation to maintain consistent training dynamics.
The Core Problem
When you use gradient accumulation, you're effectively increasing your batch size. If you accumulate gradients over 4 steps with a physical batch size of 8, your effective batch size is 32. This changes how your optimizer behaves.
Here's the key insight: larger batch sizes require larger learning rates to maintain the same training progress per example seen. If you double your effective batch size but keep the learning rate the same, your model will train more conservatively—taking smaller steps relative to the amount of data processed.
The Linear Scaling Rule
The most common approach is linear scaling: if you multiply your effective batch size by N, multiply your learning rate by N as well.
Example:
- Original: batch size 16, learning rate 0.001
- With 4x accumulation: effective batch size 64
- Scaled learning rate: 0.001 × 4 = 0.004
This rule works because gradient updates should be proportional to the amount of data processed. Without scaling, you're making the same-sized weight updates despite seeing 4x more data—essentially under-stepping.
When to Scale
Scale your learning rate when:
- Changing accumulation steps during the same experiment
- Comparing runs with different accumulation settings
- Moving from single-GPU to distributed training with accumulation
Important Caveats
The linear rule is a starting point, not gospel. Very large batch sizes may need sub-linear scaling (e.g., square-root scaling) or warmup periods to avoid instability.
Key Takeaway: When using gradient accumulation, scale your learning rate proportionally to the increase in effective batch size to maintain equivalent training dynamics—doubling effective batch size typically means doubling learning rate.