Course contentsShow
← Machine Learning and Deep Learning
Lesson 2709 of 3,53859. Distributed Training: Data ParallelismFree lesson

Effective Batch Size in Data Parallelism

How effective batch size scales with the number of workers and implications for learning rate scaling.

Effective Batch Size in Data Parallelism

You'll learn: How distributing training across multiple workers changes your actual batch size and why this matters for learning dynamics.

The Core Concept

When you run data parallel training across multiple workers, each worker processes its own mini-batch independently. The effective batch size is the total amount of data processed before gradients are averaged and weights are updated — it's the sum of all workers' local batch sizes.

If you train on 4 GPUs, each with a local batch size of 32, your effective batch size isn't 32 — it's 128 (4 × 32).

Why This Matters

Larger batch sizes fundamentally change training dynamics:

  • Fewer parameter updates per epoch: With 4× the batch size, you make 1/4 as many gradient steps through your dataset
  • Different gradient noise: Larger batches produce more stable, lower-variance gradient estimates
  • Convergence behavior changes: The optimization landscape looks "smoother" with less stochastic exploration

The Learning Rate Scaling Rule

To maintain similar convergence behavior when scaling up workers, you typically need to scale your learning rate proportionally. This is called linear scaling:

new_learning_rate = base_learning_rate × num_workers

Why? With a 4× larger batch, each gradient step covers 4× more ground in parameter space. A proportionally larger learning rate compensates, keeping the effective step size similar.

Important caveat: This rule works best with warmup and may need adjustment for very large batch sizes (thousands).

Key Takeaway: In data parallelism, your effective batch size equals local_batch_size × num_workers, and you should typically scale your learning rate proportionally to maintain training dynamics.