What is Mixture of Experts?
You'll learn: How Mixture of Experts (MoE) scales model capacity by activating only a subset of specialized neural network modules per input, dramatically reducing compute costs.
The Core Concept
Imagine a hospital where instead of one general practitioner seeing every patient, you have specialists—a cardiologist, neurologist, orthopedist, etc. Each patient is routed to the 1-2 specialists most relevant to their condition, not all doctors at once.
Mixture of Experts works the same way. Instead of passing every token through one massive feedforward network, you have multiple smaller "expert" networks. A routing mechanism (often called a "gating network") examines each token and decides which 1-2 experts should process it. The other experts remain dormant for that token.
Why This Matters for LLMs
In standard Transformers, the feedforward network (FFN) accounts for ~2/3 of model parameters. When you scale to 70B+ parameters, every token must flow through this enormous FFN—expensive!
With MoE:
- You might have 8 expert FFNs instead of 1 large FFN
- Each token activates only 2 experts (sparse activation)
- Total capacity: 8× the parameters
- Actual compute per token: Only 2× (since only 2/8 experts run)
This lets you build models with 100B+ parameters that cost as much to run as 13B dense models.
The Trade-off
You gain parameter efficiency but face new challenges: routing decisions must be learned, load balancing across experts is critical, and memory requirements remain high (all expert weights must still be loaded).
Key Takeaway: Mixture of Experts achieves massive model scaling by routing each input to specialized subnetworks, activating only a fraction of the total parameters per forward pass—unlocking greater capacity without proportional computational cost.