Model Parallelism vs Data Parallelism
What you'll learn: You'll understand the fundamental difference between splitting your model across devices versus splitting your data, and when each strategy is essential.
The Core Distinction
Data parallelism replicates your entire model on each GPU and splits the training data across workers. Each GPU processes a different batch, computes gradients, and those gradients are averaged (via all-reduce operations you've learned). This is what DDP, FSDP, and ZeRO optimize—they're all flavors of data parallelism, varying in how they handle model state.
Model parallelism splits the model itself across multiple GPUs. Different layers or components live on different devices, and activations flow from one GPU to the next during forward and backward passes.
When Model Parallelism Becomes Necessary
Imagine a transformer with 175 billion parameters. Even with ZeRO Stage 3 sharding optimizer states, gradients, and parameters, a single GPU's memory (typically 40-80GB) cannot hold the model. Data parallelism alone fails because every worker still needs a copy of the model's parameters during computation.
Model parallelism becomes essential when:
- Model parameters + optimizer states + activations exceed a single GPU's memory
- Even with FSDP/ZeRO sharding, the model is too large to fit
- You need different model components on different devices
Think of it like this: data parallelism is photocopying a book and having multiple readers process different pages simultaneously. Model parallelism is tearing a single book into chapters and having readers process them sequentially, passing results along.
The tradeoff? Model parallelism introduces pipeline bubbles (idle time) and sequential dependencies, while data parallelism enables true parallel computation but requires full model replicas.
Key Takeaway: Use data parallelism when your model fits in memory; switch to model parallelism when models exceed single-GPU capacity despite ZeRO/FSDP optimizations—often combined as hybrid parallelism for massive models.