Tensor Parallelism Fundamentals
What you'll learn: How tensor parallelism splits individual layers across devices by partitioning tensors along specific dimensions, enabling parallel computation within a single layer.
The Core Idea
Unlike pipeline parallelism (which splits a model vertically across layers) or data parallelism (which replicates the entire model), tensor parallelism splits individual operations horizontally—distributing the computation of a single layer across multiple devices.
Think of it like a restaurant kitchen where multiple chefs don't work on different courses (pipeline) or cook separate meals (data parallel), but instead collaborate on the same dish simultaneously—one chef dices vegetables while another prepares the sauce, then they combine results.
How Tensor Partitioning Works
In tensor parallelism, you partition weight matrices and activations along specific dimensions. For example, in a fully connected layer with weight matrix W of shape [d_in, d_out]:
- Column partitioning: Split
Walong columns into[d_in, d_out/N]chunks across N devices - Each device computes a partial output independently
- Results are gathered/reduced as needed
For matrix multiplication Y = XW, if W is column-partitioned, each device computes a slice of Y independently—no communication needed during forward pass. During backward pass, gradients flow back and require synchronization.
Why It Matters
Tensor parallelism is essential when:
- A single layer's parameters don't fit on one GPU
- You need fine-grained parallelism within layers (common in Transformers)
- You want to minimize communication by keeping activations local
Key Takeaway: Tensor parallelism partitions individual weight tensors and operations across devices, enabling parallel computation within layers rather than across layers or data samples—critical for training models where single layers exceed single-device memory.