Course contentsShow
← Machine Learning and Deep Learning
Lesson 2712 of 3,53859. Distributed Training: Data ParallelismFree lesson

Ring All-Reduce Algorithm

How ring all-reduce achieves efficient gradient synchronization with optimal bandwidth utilization across workers.

Ring All-Reduce Algorithm

You'll learn how ring all-reduce efficiently synchronizes gradients across multiple workers by organizing them in a logical ring and reducing communication bottlenecks.

The Core Idea

When training a deep learning model across multiple GPUs or machines using data parallelism, each worker computes gradients on its local data batch. These gradients must be averaged across all workers—that's the all-reduce operation you learned about earlier. But how do we do this efficiently?

The ring all-reduce algorithm solves this by arranging workers in a logical ring topology. Instead of having one central node collect and broadcast all gradients (which creates a bottleneck), workers pass data to their neighbors in the ring through two phases:

Reduce-Scatter Phase

Workers divide their gradient tensors into N chunks (where N = number of workers). In N-1 steps, each worker receives a chunk from its left neighbor, adds it to its own corresponding chunk, and sends the result to its right neighbor. After this phase, each worker has the complete sum for one chunk.

All-Gather Phase

In another N-1 steps, workers pass their fully-reduced chunks around the ring. Each worker receives complete chunks and forwards them to the right neighbor. After N-1 steps, every worker has all reduced chunks—the complete averaged gradient.

Why It's Optimal

The beauty of ring all-reduce is bandwidth optimality: each worker sends and receives exactly 2(N-1)/N times the data size, approaching 2× as N grows. No single worker becomes a bottleneck, and all network links are utilized simultaneously. This is why frameworks like Horovod use it for efficient distributed training.

Key Takeaway: Ring all-reduce achieves efficient gradient synchronization by organizing workers in a ring and splitting the all-reduce into reduce-scatter and all-gather phases, ensuring optimal bandwidth utilization without communication bottlenecks.