Routing Mechanisms in MoE
What you'll learn: How gating networks intelligently decide which expert(s) should process each input token in a Mixture of Experts architecture.
The Core Problem
In a Mixture of Experts (MoE) model, you have multiple specialized "expert" networks, but you can't afford to run every token through every expert—that would explode your compute costs. Instead, you need a smart router (also called a gating network) that decides which expert(s) should handle each token.
How Routing Works
The gating network is a lightweight learned module—typically just a simple linear layer followed by a softmax—that takes a token's representation as input and outputs a probability distribution over all available experts.
Think of it like a hospital triage nurse: each patient (token) arrives, and the nurse quickly assesses them and routes them to the appropriate specialist (expert) rather than having every doctor examine every patient.
The Routing Decision
For each token independently:
- The gating network computes a score for every expert
- These scores are converted to probabilities (via softmax)
- The top-k experts with highest probabilities are selected (commonly k=1 or k=2)
- The token is sent only to those chosen experts
- The expert outputs are combined using the gating probabilities as weights
This learned, token-level routing means different tokens can specialize—perhaps syntax-related tokens go to one expert while domain-specific terms go to another.
Why It Matters
Dynamic routing is what makes MoE efficient: you get model capacity from many experts, but only activate a small subset per token, keeping inference costs manageable while maintaining model quality.
Key Takeaway: Gating networks enable sparse activation in MoE by learning to route each token to the most relevant expert(s), balancing computational efficiency with specialization.