Course contentsShow
← Machine Learning and Deep Learning
Lesson 2766 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Inter-Node Communication Challenges

Understand bandwidth bottlenecks and latency issues when model parallelism spans multiple nodes in a cluster.

Inter-Node Communication Challenges

What you'll learn: You'll understand how bandwidth and latency become critical bottlenecks when model parallelism spans multiple machines in a distributed cluster.

The Core Problem

When you split a large model across multiple nodes (separate physical machines) using pipeline or tensor parallelism, those nodes must constantly exchange data: activations flowing forward, gradients flowing backward, and sometimes intermediate tensors for synchronization. Unlike multi-GPU communication within a single node (which happens over fast NVLink or PCIe), inter-node communication travels over the network—typically Ethernet or InfiniBand—which is orders of magnitude slower.

Why This Matters for Model Parallelism

Bandwidth Bottlenecks

Network bandwidth limits how much data you can transfer per second. When your tensor parallelism requires frequent all-reduce operations or your pipeline stages send large activation tensors between nodes, the network becomes saturated. Your GPUs sit idle waiting for data to arrive, wasting expensive compute time.

Latency Issues

Even with sufficient bandwidth, latency—the time for a single message to travel between nodes—adds up. In tensor parallelism, where each transformer layer might require multiple round-trip communications, even milliseconds of latency per hop multiply across hundreds of layers. Pipeline parallelism with small microbatches suffers similarly: each stage waits for the previous node's output before proceeding.

The Analogy

Imagine building a car on an assembly line where each station is in a different city. Even if trucks (bandwidth) can carry large loads, the travel time (latency) between cities makes the whole process painfully slow compared to stations in the same factory.

Key Takeaway: Inter-node communication introduces bandwidth and latency constraints that can severely limit the scalability of model parallelism; minimizing cross-node data transfers and choosing fast interconnects (InfiniBand over Ethernet) are critical for efficient distributed training.