Gradient Averaging Across Workers
What you'll learn: Why distributed training requires averaging (not summing) gradients from multiple workers to preserve the mathematical properties of single-GPU training.
The Core Problem
When you split a batch across multiple workers in data parallelism, each worker computes gradients on its local mini-batch. To update the model correctly, you need to combine these gradients—but how you combine them matters tremendously.
Why Average, Not Sum?
Imagine training on a batch of 128 samples. On a single GPU, you'd:
- Compute loss for all 128 samples
- Calculate gradients (automatically averaged over the batch)
- Update weights
Now split that same 128 samples across 4 workers (32 samples each). Each worker computes gradients for its 32 samples. If you simply summed these gradients, you'd get gradients 4× larger than the single-GPU case—leading to 4× larger weight updates and unstable training.
Averaging the gradients across workers gives you the exact same update as if you'd processed all 128 samples on one GPU. It's mathematically equivalent because:
gradient_avg = (grad_worker1 + grad_worker2 + grad_worker3 + grad_worker4) / 4
This preserves the expected gradient magnitude and ensures your learning rate, momentum, and other hyperparameters behave identically.
The Analogy
Think of measuring average rainfall across a city. You place 4 rain gauges in different locations. To get the city average, you sum the readings and divide by 4—not just add them up, which would give you 4× the actual rainfall.
Key Takeaway: Gradient averaging (dividing the sum by the number of workers) maintains mathematical equivalence to single-GPU training, ensuring stable and predictable model updates across distributed workers.