Effective Batch Size in Data Parallelism
You'll learn: How distributing training across multiple workers changes your actual batch size and why this matters for learning dynamics.
The Core Concept
When you run data parallel training across multiple workers, each worker processes its own mini-batch independently. The effective batch size is the total amount of data processed before gradients are averaged and weights are updated — it's the sum of all workers' local batch sizes.
If you train on 4 GPUs, each with a local batch size of 32, your effective batch size isn't 32 — it's 128 (4 × 32).
Why This Matters
Larger batch sizes fundamentally change training dynamics:
- Fewer parameter updates per epoch: With 4× the batch size, you make 1/4 as many gradient steps through your dataset
- Different gradient noise: Larger batches produce more stable, lower-variance gradient estimates
- Convergence behavior changes: The optimization landscape looks "smoother" with less stochastic exploration
The Learning Rate Scaling Rule
To maintain similar convergence behavior when scaling up workers, you typically need to scale your learning rate proportionally. This is called linear scaling:
new_learning_rate = base_learning_rate × num_workers
Why? With a 4× larger batch, each gradient step covers 4× more ground in parameter space. A proportionally larger learning rate compensates, keeping the effective step size similar.
Important caveat: This rule works best with warmup and may need adjustment for very large batch sizes (thousands).
Key Takeaway: In data parallelism, your effective batch size equals local_batch_size × num_workers, and you should typically scale your learning rate proportionally to maintain training dynamics.