Course contentsShow
← AI Engineering
Lesson 1074 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Tensor Parallelism Fundamentals

How individual layers are split across GPUs, enabling models larger than single GPU memory.

Tensor Parallelism Fundamentals

What you'll learn: How tensor parallelism splits individual neural network layers across multiple GPUs to run models that are too large for a single GPU's memory.

The Core Problem

When a model (like a 70B parameter LLM) exceeds a single GPU's memory capacity, you need a way to distribute it. Tensor parallelism solves this by splitting the weight matrices within each layer across multiple GPUs, allowing them to work together on the same operation.

How It Works

Think of a neural network layer like a massive spreadsheet calculation. Instead of one person doing all the math, tensor parallelism divides the spreadsheet columns among multiple workers who calculate their portion simultaneously, then combine results.

Here's what happens in practice:

Layer-Level Splitting

Each transformer layer contains large weight matrices (for attention and feed-forward networks). Tensor parallelism partitions these matrices across GPUs:

  • Attention layers: Split the query, key, and value projection matrices
  • Feed-forward layers: Split the first linear transformation across GPUs
  • Each GPU computes its portion using the same input
  • Results are combined (via all-reduce or concatenation) before moving to the next layer

Communication Pattern

Unlike data parallelism (where each GPU processes different inputs), tensor parallelism requires GPUs to communicate during the forward pass of each layer. This means high-speed interconnects (NVLink, InfiniBand) are critical for performance.

Why It Matters

Tensor parallelism enables you to run massive models that would be impossible otherwise. A 70B model might need 140GB of memory in FP16, but split across 4 GPUs, each only needs ~35GB.

Key Takeaway: Tensor parallelism splits individual layer weights across GPUs horizontally, enabling models larger than single-GPU memory by distributing computation within each layer rather than across layers.