Course contentsShow
← Machine Learning and Deep Learning
Lesson 3004 of 3,53865. LLM Inference EnginesFree lesson

Model Sharding and Tensor Parallelism for Serving

Split large models across GPUs using tensor parallelism to serve models too large for single devices.

Model Sharding and Tensor Parallelism for Serving

What you'll learn: How to split massive models across multiple GPUs using tensor parallelism so you can serve models too large to fit on a single device.

The Challenge: Models Bigger Than GPU Memory

Modern LLMs often exceed the memory capacity of a single GPU. A 70B parameter model might need 140GB in FP16, but consumer GPUs max out at 24-80GB. You need a way to distribute the model itself across devices.

Tensor Parallelism: Splitting Weights, Not Batches

Unlike data parallelism (where each GPU holds the full model and processes different data), tensor parallelism splits individual weight matrices across GPUs.

Think of it like a restaurant kitchen: instead of having multiple identical stations (data parallelism), you have one station where different chefs simultaneously work on different parts of the same large dish.

How It Works

For a transformer layer's linear operation Y = XW:

  • Split weight matrix W column-wise across N GPUs
  • Each GPU computes its portion: Y_i = X @ W_i
  • Aggregate results (sum/concatenate) as needed

Attention heads are natural parallelization boundaries—split 32 heads across 4 GPUs (8 heads each), and each GPU independently computes its subset before the final projection combines results.

Communication Overhead

GPUs must exchange partial results via all-reduce or all-gather operations. Fast interconnects (NVLink, InfiniBand) are critical—slow communication negates the benefits.

Practical Considerations

  • Memory efficiency: Enables serving models impossible on single GPUs
  • Latency cost: Inter-GPU communication adds microseconds-to-milliseconds per layer
  • Sweet spot: Use when model size demands it, not for small models where overhead dominates

Key Takeaway: Tensor parallelism splits weight matrices across GPUs to serve models too large for single devices, trading communication overhead for the ability to run massive LLMs in production.