Megatron-LM Column and Row Parallelism
What you'll learn: How Megatron-LM splits transformer layers across GPUs by dividing weight matrices into columns and rows, enabling massive models to fit in distributed memory.
The Core Idea
Megatron-LM implements tensor parallelism by strategically partitioning the large weight matrices inside transformer blocks. Instead of replicating entire layers on each GPU, it divides them along specific dimensions:
- Column parallelism: Splits weight matrices vertically (by output features)
- Row parallelism: Splits weight matrices horizontally (by input features)
How It Works in Transformers
Feed-Forward Network (FFN)
The FFN has two linear layers. Megatron splits them as follows:
-
First linear layer (expand): Uses column parallelism. Each GPU holds a portion of the output features. The input is replicated, each GPU computes its slice independently, and outputs are kept partitioned.
-
Second linear layer (project back): Uses row parallelism. Each GPU holds a portion of the input features (matching the previous layer's split). Results are reduced (summed) across GPUs to produce the final output.
Multi-Head Attention
The query, key, and value projection matrices use column parallelism — each GPU computes attention for a subset of heads. The output projection uses row parallelism, followed by an all-reduce to combine results.
The Beautiful Symmetry
Column parallelism naturally feeds into row parallelism with minimal communication: only one all-reduce per transformer sub-layer (after the row-parallel operation). This keeps communication overhead low while distributing both computation and memory.
Key Takeaway: Megatron-LM uses column parallelism for expanding projections and row parallelism for reducing projections, creating an efficient pattern that splits transformer weights across GPUs with just two all-reduce operations per layer (one after attention, one after FFN).