Course contentsShow
← Machine Learning and Deep Learning
Lesson 2760 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Tensor Parallelism Fundamentals

Understand how tensor parallelism splits individual layers across devices by partitioning tensors along specific dimensions.

Tensor Parallelism Fundamentals

What you'll learn: How tensor parallelism splits individual layers across devices by partitioning tensors along specific dimensions, enabling parallel computation within a single layer.

The Core Idea

Unlike pipeline parallelism (which splits a model vertically across layers) or data parallelism (which replicates the entire model), tensor parallelism splits individual operations horizontally—distributing the computation of a single layer across multiple devices.

Think of it like a restaurant kitchen where multiple chefs don't work on different courses (pipeline) or cook separate meals (data parallel), but instead collaborate on the same dish simultaneously—one chef dices vegetables while another prepares the sauce, then they combine results.

How Tensor Partitioning Works

In tensor parallelism, you partition weight matrices and activations along specific dimensions. For example, in a fully connected layer with weight matrix W of shape [d_in, d_out]:

  • Column partitioning: Split W along columns into [d_in, d_out/N] chunks across N devices
  • Each device computes a partial output independently
  • Results are gathered/reduced as needed

For matrix multiplication Y = XW, if W is column-partitioned, each device computes a slice of Y independently—no communication needed during forward pass. During backward pass, gradients flow back and require synchronization.

Why It Matters

Tensor parallelism is essential when:

  • A single layer's parameters don't fit on one GPU
  • You need fine-grained parallelism within layers (common in Transformers)
  • You want to minimize communication by keeping activations local

Key Takeaway: Tensor parallelism partitions individual weight tensors and operations across devices, enabling parallel computation within layers rather than across layers or data samples—critical for training models where single layers exceed single-device memory.