Course contentsShow
← Machine Learning and Deep Learning
Lesson 2762 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Communication Patterns in Tensor Parallelism

Understand all-reduce and all-gather communication operations required for tensor parallel computation.

Communication Patterns in Tensor Parallelism

You'll learn: How tensor parallel models communicate intermediate results across GPUs during forward and backward passes.

The Communication Challenge

When you split tensors (weights, activations) across multiple GPUs using column and row parallelism (Megatron-LM style), each GPU holds only part of the computation. To produce correct outputs, GPUs must exchange partial results at specific points in the network.

Think of it like a relay race where each runner completes one leg: you need handoff points where information passes between teammates.

All-Gather: Collecting the Full Picture

All-gather collects partial tensors from all GPUs and distributes the complete, concatenated result to everyone.

Example: In column-parallel linear layers, each GPU computes part of the output features. Before the next layer can use them, all GPUs need the full activation tensor. All-gather assembles these pieces.

Analogy: Each person has one page of a document; all-gather makes copies of the complete document for everyone.

All-Reduce: Summing Contributions

All-reduce sums tensors across all GPUs and gives each GPU the total.

Example: In row-parallel layers, each GPU computes a partial matrix multiplication. The final output requires summing these partial results. All-reduce efficiently combines them.

Analogy: Each person counts votes in their district; all-reduce calculates the total and tells everyone the final count.

When They Happen

  • Forward pass: All-gather after column-parallel layers; all-reduce after row-parallel layers
  • Backward pass: The pattern reverses—all-reduce where forward had all-gather, and vice versa (following backpropagation chain rule)

These operations occur at layer boundaries, creating predictable communication patterns that overlap with computation when optimized.

Key Takeaway: Tensor parallelism relies on all-gather to reconstruct split tensors and all-reduce to combine partial computations, with communication patterns reversing between forward and backward passes.