Course contentsShow
← Machine Learning and Deep Learning
Lesson 2714 of 3,53859. Distributed Training: Data ParallelismFree lesson

Scaling Efficiency and Strong vs Weak Scaling

Measuring speedup: strong scaling (fixed problem size) vs weak scaling (problem size grows with workers).

Scaling Efficiency and Strong vs Weak Scaling

What you'll learn: How to measure whether your distributed training actually gets faster as you add more workers, using two key scaling paradigms.

Why Scaling Efficiency Matters

Adding more GPUs to your distributed training doesn't always mean proportional speedups. If 4 GPUs don't train 4× faster than 1 GPU, something's eating your gains—usually communication overhead, gradient synchronization, or idle time. Scaling efficiency measures how well your speedup matches the ideal case.

Strong Scaling: Fixed Problem, More Workers

Strong scaling keeps your total problem size constant while adding workers. If your batch size is 256 and you scale from 1 to 4 GPUs, each GPU now processes 64 samples instead of 256.

Ideal scenario: 4 GPUs = 4× speedup
Reality: You get 3.2× speedup due to communication overhead

Think of it like four people sharing one pizza-making task: the pizza size doesn't change, but coordination (communication) becomes a bottleneck.

Weak Scaling: Growing Problem with Workers

Weak scaling increases the problem size proportionally with workers. With 4× the GPUs, you use 4× the batch size (from 256 to 1024), so each GPU still processes 256 samples.

Ideal scenario: Time per iteration stays constant
Reality: Time increases slightly as communication grows

This is like giving each person their own pizza to make—everyone stays busy, but occasionally they need to sync their recipes (gradients).

Which Matters for You?

  • Strong scaling reveals pure parallelization efficiency—can you train your existing workload faster?
  • Weak scaling shows throughput capacity—can you handle bigger batches without slowdown?

Most deep learning emphasizes weak scaling: larger batches often improve generalization (with proper learning rate scaling), and throughput matters more than wall-clock time for large models.

Key Takeaway: Strong scaling measures speedup for fixed batch sizes; weak scaling measures sustained efficiency as you scale both batch size and workers together—understanding both reveals where communication costs hurt your distributed training.