Course contentsShow
← Machine Learning and Deep Learning
Lesson 2765 of 3,53860. Distributed Training: Model Parallelism and Mixed PrecisionFree lesson

Expert Parallelism for MoE Models

Learn how mixture-of-experts models distribute expert computation across devices for efficient sparse model training.

Expert Parallelism for MoE Models

What you'll learn: How mixture-of-experts (MoE) models distribute individual experts across devices to enable efficient training of massive sparse models.

Core Idea

In mixture-of-experts architectures, different "expert" sub-networks specialize in handling different parts of the input space. Rather than activating all parameters for every input (like dense models), MoE models route each input to only a few experts—making them conditionally sparse.

Expert parallelism places each expert (or group of experts) on different GPUs or devices. When an input token needs processing, it's dynamically routed to the device holding the appropriate expert, computation happens there, and results return to the original device.

Why It's Different

Unlike tensor or pipeline parallelism that split layers, expert parallelism splits the experts within a layer. Each device holds complete copies of non-expert layers (self-attention, feed-forward) but only a subset of experts.

The Routing Challenge

A gating network decides which expert(s) each token should use. This creates uneven load balancing—some experts may receive many tokens while others get few. Expert parallelism must handle:

  • All-to-all communication: Tokens scatter from all devices to expert devices, then gather back
  • Capacity constraints: Limiting tokens per expert to prevent memory overflow
  • Load balancing: Encouraging the router to use experts evenly

Analogy

Think of a hospital with specialists (cardiologist, neurologist, orthopedist) distributed across different floors. Each patient is routed to the right specialist's floor, treated there, then returns to the main desk—not every specialist sees every patient.

Key Takeaway: Expert parallelism distributes MoE experts across devices with dynamic all-to-all routing, enabling training of models with trillions of parameters where only a sparse subset activates per input.