Communication Overhead and Bandwidth
You'll learn why moving data between GPUs can become the bottleneck in multi-GPU LLM deployments, and how to minimize that cost.
The Core Problem
When you split a large language model across multiple GPUs (using tensor or pipeline parallelism), those GPUs must constantly talk to each other. In tensor parallelism, every forward pass requires exchanging activation tensors. In pipeline parallelism, intermediate outputs move between stages. This communication overhead — the time spent sending data instead of computing — can easily dominate your inference latency if not managed carefully.
Think of it like a team of chefs working on different parts of a meal. If they're in separate kitchens and need to constantly pass ingredients back and forth through a narrow hallway, they'll spend more time walking than cooking.
Bandwidth Matters
The bandwidth between GPUs determines how fast data moves. Standard PCIe connections offer 16-32 GB/s, which sounds fast but becomes a bottleneck when transferring multi-gigabyte tensors dozens of times per inference request.
NVLink to the Rescue
NVLink is NVIDIA's high-speed interconnect technology, providing 300-600 GB/s bandwidth between GPUs (10-20× faster than PCIe). When planning multi-GPU deployments, NVLink-equipped systems dramatically reduce communication overhead, making parallelism practical for large models.
Common Bottlenecks
- All-reduce operations in tensor parallelism synchronize gradients/activations across all GPUs
- Point-to-point transfers in pipeline parallelism create sequential dependencies
- Small batch sizes worsen the compute-to-communication ratio (more time waiting than working)
Key Takeaway: Communication overhead scales with model parallelism degree and can erase performance gains — prioritize NVLink hardware and larger batch sizes to keep GPUs computing rather than waiting.