Network Topology and Bandwidth Considerations
What You'll Learn
You'll understand how network architecture, speed, and latency critically impact the efficiency of multi-node distributed training.
Why Network Infrastructure Matters
When training large models across multiple nodes, network performance becomes a primary bottleneck. Unlike single-node multi-GPU setups where GPUs communicate through fast PCIe or NVLink connections, multi-node training requires data to traverse physical network cables between machines.
Think of it like shipping packages: sending something across your house (intra-node) is instant, but shipping between cities (inter-node) depends heavily on the courier service quality.
Key Network Factors
Bandwidth
The amount of data that can be transferred per second. Common options:
- Ethernet: 10-100 Gbps (standard datacenter)
- InfiniBand: 200-400 Gbps (high-performance computing)
Higher bandwidth means gradients and parameters can be synchronized faster during all-reduce operations.
Latency
The time delay before data transfer begins. InfiniBand typically offers ~1-2 microseconds vs Ethernet's ~10-50 microseconds. Low latency is crucial when synchronizing small, frequent messages.
Network Topology
How nodes are physically connected:
- Fat-tree topology: Common in datacenters, provides multiple paths between nodes
- Direct connections: InfiniBand often uses direct node-to-node links
Poor topology creates congestion—imagine all-reduce operations where every GPU must communicate with every other GPU. If nodes share limited network switches, they compete for bandwidth.
Practical Impact
Operations like all-reduce in DDP or all-gather in FSDP transfer gigabytes of data per training step. With slow networks, GPUs spend more time waiting than computing, destroying scaling efficiency. This is why gradient accumulation and careful batch sizing matter—they reduce communication frequency.
Key Takeaway: Network bandwidth and topology directly determine whether multi-node training scales efficiently; InfiniBand's superior speed and low latency make it the gold standard for large-scale distributed training, while standard Ethernet may bottleneck gradient synchronization.