Course contentsShow
← Machine Learning and Deep Learning
Lesson 2707 of 3,53859. Distributed Training: Data ParallelismFree lesson

All-Reduce Operation Fundamentals

The all-reduce collective operation that synchronizes gradients across all devices efficiently.

All-Reduce Operation Fundamentals

What you'll learn: How the all-reduce collective operation efficiently synchronizes gradients across multiple devices in distributed training.

The Core Challenge

When training with data parallelism, each worker computes gradients on its local data batch. Before updating model weights, all workers need identical gradient values—the average of everyone's gradients. The naive approach of sending all gradients to one machine, summing them, then broadcasting back creates a bottleneck. All-reduce solves this elegantly.

What All-Reduce Does

All-reduce is a collective communication operation where:

  1. Every worker starts with a local gradient tensor
  2. The operation simultaneously reduces (sums) all copies across workers
  3. Every worker ends with the same result—the sum (or average) of all inputs

Think of it like a group of accountants, each with partial expense reports. Instead of one person collecting all reports sequentially, they simultaneously exchange information in a structured pattern until everyone has the complete total.

Why It's Efficient

All-reduce uses clever communication patterns (like ring or tree topologies) to:

  • Parallelize communication across all workers
  • Balance network load—no single machine becomes a bottleneck
  • Minimize total data transferred

For example, in a ring all-reduce, workers form a logical ring and pass chunks of data to their neighbors in coordinated rounds. Each worker simultaneously sends and receives, utilizing all network links.

In Practice

After computing local gradients, a single all-reduce operation gives every worker the averaged gradients. Then all workers perform identical weight updates, keeping models synchronized.

Key Takeaway: All-reduce is the fundamental primitive that makes data parallelism scalable by efficiently computing and distributing the sum of gradients across all workers without creating communication bottlenecks.