Course contentsShow
← Machine Learning and Deep Learning
Lesson 2793 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Network Topology and Bandwidth Considerations

Impact of network speed, latency, and topology (InfiniBand, Ethernet) on multi-node training performance.

Network Topology and Bandwidth Considerations

What You'll Learn

You'll understand how network architecture, speed, and latency critically impact the efficiency of multi-node distributed training.

Why Network Infrastructure Matters

When training large models across multiple nodes, network performance becomes a primary bottleneck. Unlike single-node multi-GPU setups where GPUs communicate through fast PCIe or NVLink connections, multi-node training requires data to traverse physical network cables between machines.

Think of it like shipping packages: sending something across your house (intra-node) is instant, but shipping between cities (inter-node) depends heavily on the courier service quality.

Key Network Factors

Bandwidth

The amount of data that can be transferred per second. Common options:

  • Ethernet: 10-100 Gbps (standard datacenter)
  • InfiniBand: 200-400 Gbps (high-performance computing)

Higher bandwidth means gradients and parameters can be synchronized faster during all-reduce operations.

Latency

The time delay before data transfer begins. InfiniBand typically offers ~1-2 microseconds vs Ethernet's ~10-50 microseconds. Low latency is crucial when synchronizing small, frequent messages.

Network Topology

How nodes are physically connected:

  • Fat-tree topology: Common in datacenters, provides multiple paths between nodes
  • Direct connections: InfiniBand often uses direct node-to-node links

Poor topology creates congestion—imagine all-reduce operations where every GPU must communicate with every other GPU. If nodes share limited network switches, they compete for bandwidth.

Practical Impact

Operations like all-reduce in DDP or all-gather in FSDP transfer gigabytes of data per training step. With slow networks, GPUs spend more time waiting than computing, destroying scaling efficiency. This is why gradient accumulation and careful batch sizing matter—they reduce communication frequency.

Key Takeaway: Network bandwidth and topology directly determine whether multi-node training scales efficiently; InfiniBand's superior speed and low latency make it the gold standard for large-scale distributed training, while standard Ethernet may bottleneck gradient synchronization.