Introduction to Model Parallelism
What you'll learn: You'll understand why massive language models can't fit on a single GPU and how model parallelism differs fundamentally from data parallelism.
The Problem: Models Bigger Than Memory
Remember from previous lessons that large models have enormous memory requirements. A 70B parameter model in FP16 precision needs roughly 140GB of memory—far exceeding what even high-end datacenter GPUs offer (typically 40-80GB). This is where model parallelism becomes essential.
Two Different Strategies
Data parallelism replicates the entire model across multiple GPUs. Each GPU processes different batches of data simultaneously but holds a complete copy of the model. This works great for scaling throughput when your model fits on one device.
Model parallelism splits the model itself across multiple GPUs. Different parts of the model live on different devices, and data flows between them during inference or training. This is necessary when a single model simply won't fit in one GPU's memory.
The Analogy
Think of an assembly line for building cars. With data parallelism, you build multiple identical assembly lines side-by-side—more cars, same design. With model parallelism, you split one assembly line across multiple stations—the engine station, the body station, etc. A single car passes through each station in sequence.
When You Need Model Parallelism
- Your model parameters exceed single-GPU memory (even with quantization)
- You want to serve models like Llama 70B, Mixtral 8x7B, or GPT-175B
- Single-GPU inference is too slow even with optimization
Key Takeaway: Model parallelism splits a single model across multiple GPUs when it's too large to fit on one device, while data parallelism replicates the entire model to process more requests in parallel.