Model Sharding and Tensor Parallelism for Serving
What you'll learn: How to split massive models across multiple GPUs using tensor parallelism so you can serve models too large to fit on a single device.
The Challenge: Models Bigger Than GPU Memory
Modern LLMs often exceed the memory capacity of a single GPU. A 70B parameter model might need 140GB in FP16, but consumer GPUs max out at 24-80GB. You need a way to distribute the model itself across devices.
Tensor Parallelism: Splitting Weights, Not Batches
Unlike data parallelism (where each GPU holds the full model and processes different data), tensor parallelism splits individual weight matrices across GPUs.
Think of it like a restaurant kitchen: instead of having multiple identical stations (data parallelism), you have one station where different chefs simultaneously work on different parts of the same large dish.
How It Works
For a transformer layer's linear operation Y = XW:
- Split weight matrix
Wcolumn-wise across N GPUs - Each GPU computes its portion:
Y_i = X @ W_i - Aggregate results (sum/concatenate) as needed
Attention heads are natural parallelization boundaries—split 32 heads across 4 GPUs (8 heads each), and each GPU independently computes its subset before the final projection combines results.
Communication Overhead
GPUs must exchange partial results via all-reduce or all-gather operations. Fast interconnects (NVLink, InfiniBand) are critical—slow communication negates the benefits.
Practical Considerations
- Memory efficiency: Enables serving models impossible on single GPUs
- Latency cost: Inter-GPU communication adds microseconds-to-milliseconds per layer
- Sweet spot: Use when model size demands it, not for small models where overhead dominates
Key Takeaway: Tensor parallelism splits weight matrices across GPUs to serve models too large for single devices, trading communication overhead for the ability to run massive LLMs in production.