Course contentsShow
← Machine Learning and Deep Learning
Lesson 1212 of 3,53827. Pretrained Language Models: GPT Family and BeyondFree lesson

Mixture of Experts in Modern GPT Architectures

How MoE routing enables efficient scaling beyond dense models in GPT-4 and similar architectures.

Mixture of Experts in Modern GPT Architectures

What you'll learn: How modern GPT models like GPT-4 use Mixture of Experts (MoE) routing to scale efficiently without making every parameter active for every token.

The Scaling Problem with Dense Models

In traditional dense models like GPT-3, every single parameter processes every single token. When you scale to 175 billion parameters, this becomes computationally expensive—you're doing massive calculations even when simpler patterns might suffice.

How Mixture of Experts Works

Think of MoE like a hospital with specialist doctors. Instead of having one general practitioner examine every patient thoroughly, a routing system (the triage nurse) quickly decides which specialist each patient needs. Most patients only see 1-2 specialists, not all 50 doctors.

In GPT architectures with MoE:

  • Each transformer layer contains multiple "expert" sub-networks (typically feed-forward networks)
  • A lightweight gating network (router) examines each token's representation
  • The router activates only top-k experts (often k=2) for that specific token
  • Different tokens can activate different experts

Why This Enables Efficient Scaling

You can have a model with trillions of total parameters, but only use a fraction per token. GPT-4 reportedly uses MoE to achieve its performance:

  • More total capacity (more experts = more specialized knowledge)
  • Lower compute per token (only active experts compute)
  • Faster inference than an equivalent dense model
  • Each expert can specialize (one for code, one for scientific text, etc.)

The router learns which expert is best for each context automatically during training.

Key Takeaway: MoE routing lets modern GPT models scale to massive parameter counts while keeping computational costs manageable by activating only relevant expert sub-networks for each token, rather than running all parameters densely.