Course contentsShow
← Machine Learning and Deep Learning
Lesson 2711 of 3,53859. Distributed Training: Data ParallelismFree lesson

Communication Overhead and Bottlenecks

Understanding that gradient synchronization introduces communication costs that can limit scaling efficiency.

Communication Overhead and Bottlenecks

What you'll learn: How gradient synchronization can become the limiting factor in scaling distributed training across multiple workers.

The Hidden Cost of Parallelism

When you distribute training across multiple GPUs or machines using data parallelism, each worker computes gradients on its local batch, then all gradients must be synchronized (typically via All-Reduce) before updating model parameters. This synchronization isn't free—it requires data transfer over network connections or even GPU interconnects.

The time spent communicating can actually exceed the time spent computing gradients, especially when:

  • Model size is large: More parameters = more gradient data to transfer
  • Network bandwidth is limited: Slow connections bottleneck the All-Reduce operation
  • Worker count increases: More participants in the All-Reduce means more coordination complexity
  • Computation is fast: Modern GPUs compute so quickly that communication becomes the dominant cost

The Scaling Efficiency Problem

Ideally, doubling your workers should halve training time. In practice, communication overhead means you get diminishing returns. If gradient computation takes 100ms but synchronization takes 150ms per iteration, adding more workers won't help—you're communication-bound, not compute-bound.

Think of it like a team project: if everyone finishes their part in 10 minutes but then spends 30 minutes on Zoom coordinating, adding more team members just makes the coordination meeting longer and more chaotic.

This is why gradient compression, overlapping communication with computation, and high-bandwidth interconnects (like NVLink or InfiniBand) are critical for efficient large-scale training.

Key Takeaway: Communication overhead from gradient synchronization can dominate training time and limit the scalability benefits of adding more workers in data parallel training.