Course contentsShow
← Machine Learning and Deep Learning
Lesson 2761 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Megatron-LM Column and Row Parallelism

Learn how Megatron splits transformer feed-forward and attention layers using column and row parallelism strategies.

Megatron-LM Column and Row Parallelism

What you'll learn: How Megatron-LM splits transformer layers across GPUs by dividing weight matrices into columns and rows, enabling massive models to fit in distributed memory.

The Core Idea

Megatron-LM implements tensor parallelism by strategically partitioning the large weight matrices inside transformer blocks. Instead of replicating entire layers on each GPU, it divides them along specific dimensions:

  • Column parallelism: Splits weight matrices vertically (by output features)
  • Row parallelism: Splits weight matrices horizontally (by input features)

How It Works in Transformers

Feed-Forward Network (FFN)

The FFN has two linear layers. Megatron splits them as follows:

  1. First linear layer (expand): Uses column parallelism. Each GPU holds a portion of the output features. The input is replicated, each GPU computes its slice independently, and outputs are kept partitioned.

  2. Second linear layer (project back): Uses row parallelism. Each GPU holds a portion of the input features (matching the previous layer's split). Results are reduced (summed) across GPUs to produce the final output.

Multi-Head Attention

The query, key, and value projection matrices use column parallelism — each GPU computes attention for a subset of heads. The output projection uses row parallelism, followed by an all-reduce to combine results.

The Beautiful Symmetry

Column parallelism naturally feeds into row parallelism with minimal communication: only one all-reduce per transformer sub-layer (after the row-parallel operation). This keeps communication overhead low while distributing both computation and memory.

Key Takeaway: Megatron-LM uses column parallelism for expanding projections and row parallelism for reducing projections, creating an efficient pattern that splits transformer weights across GPUs with just two all-reduce operations per layer (one after attention, one after FFN).