Broadcast and Reduce Operations
What you'll learn: How collective communication primitives enable efficient parameter synchronization across distributed workers.
The Core Idea
In distributed training, workers need to share information—initial model weights, gradients, and other tensors. Broadcast and reduce are fundamental collective operations that make this possible:
- Broadcast: One worker (typically rank 0) sends a tensor to all other workers. Everyone receives an identical copy.
- Reduce: All workers send their tensors to one destination worker, which combines them using an operation like sum or average.
Think of broadcast like a teacher announcing homework to the entire class—one source, many receivers. Reduce is like collecting test scores from all students to calculate the class average—many sources, one receiver.
When You Use Each
Broadcast happens at the start of training to ensure all workers begin with identical model parameters. Without it, each worker would start with random, different weights.
Reduce is less common alone because in data parallelism you typically need the combined result everywhere, not just on one worker. That's where all_reduce (which you've seen) comes in—it's essentially a reduce followed by a broadcast, giving every worker the aggregated result.
Why They Matter
These primitives are building blocks:
broadcast: Initialization and synchronizationreduce: Gather results to one location (useful for metrics)all_reduce: Gradient synchronization (reduce + broadcast combined)
Understanding broadcast and reduce helps you reason about communication patterns, debug synchronization issues, and optimize distributed workflows.
Key Takeaway: Broadcast distributes data from one worker to all, while reduce aggregates data from all workers to one; together they form the foundation for efficient collective communication in distributed training.