Scaling Efficiency and Strong vs Weak Scaling
What you'll learn: How to measure whether your distributed training actually gets faster as you add more workers, using two key scaling paradigms.
Why Scaling Efficiency Matters
Adding more GPUs to your distributed training doesn't always mean proportional speedups. If 4 GPUs don't train 4× faster than 1 GPU, something's eating your gains—usually communication overhead, gradient synchronization, or idle time. Scaling efficiency measures how well your speedup matches the ideal case.
Strong Scaling: Fixed Problem, More Workers
Strong scaling keeps your total problem size constant while adding workers. If your batch size is 256 and you scale from 1 to 4 GPUs, each GPU now processes 64 samples instead of 256.
Ideal scenario: 4 GPUs = 4× speedup
Reality: You get 3.2× speedup due to communication overhead
Think of it like four people sharing one pizza-making task: the pizza size doesn't change, but coordination (communication) becomes a bottleneck.
Weak Scaling: Growing Problem with Workers
Weak scaling increases the problem size proportionally with workers. With 4× the GPUs, you use 4× the batch size (from 256 to 1024), so each GPU still processes 256 samples.
Ideal scenario: Time per iteration stays constant
Reality: Time increases slightly as communication grows
This is like giving each person their own pizza to make—everyone stays busy, but occasionally they need to sync their recipes (gradients).
Which Matters for You?
- Strong scaling reveals pure parallelization efficiency—can you train your existing workload faster?
- Weak scaling shows throughput capacity—can you handle bigger batches without slowdown?
Most deep learning emphasizes weak scaling: larger batches often improve generalization (with proper learning rate scaling), and throughput matters more than wall-clock time for large models.
Key Takeaway: Strong scaling measures speedup for fixed batch sizes; weak scaling measures sustained efficiency as you scale both batch size and workers together—understanding both reveals where communication costs hurt your distributed training.