Course contentsShow
← Machine Learning and Deep Learning
Lesson 2785 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Learning Rate Scaling with Gradient Accumulation

Adjusting learning rates appropriately when changing accumulation steps to maintain equivalent training dynamics.

Learning Rate Scaling with Gradient Accumulation

What you'll learn: How to adjust your learning rate when using gradient accumulation to maintain consistent training dynamics.

The Core Problem

When you use gradient accumulation, you're effectively increasing your batch size. If you accumulate gradients over 4 steps with a physical batch size of 8, your effective batch size is 32. This changes how your optimizer behaves.

Here's the key insight: larger batch sizes require larger learning rates to maintain the same training progress per example seen. If you double your effective batch size but keep the learning rate the same, your model will train more conservatively—taking smaller steps relative to the amount of data processed.

The Linear Scaling Rule

The most common approach is linear scaling: if you multiply your effective batch size by N, multiply your learning rate by N as well.

Example:

  • Original: batch size 16, learning rate 0.001
  • With 4x accumulation: effective batch size 64
  • Scaled learning rate: 0.001 × 4 = 0.004

This rule works because gradient updates should be proportional to the amount of data processed. Without scaling, you're making the same-sized weight updates despite seeing 4x more data—essentially under-stepping.

When to Scale

Scale your learning rate when:

  • Changing accumulation steps during the same experiment
  • Comparing runs with different accumulation settings
  • Moving from single-GPU to distributed training with accumulation

Important Caveats

The linear rule is a starting point, not gospel. Very large batch sizes may need sub-linear scaling (e.g., square-root scaling) or warmup periods to avoid instability.

Key Takeaway: When using gradient accumulation, scale your learning rate proportionally to the increase in effective batch size to maintain equivalent training dynamics—doubling effective batch size typically means doubling learning rate.