Course contentsShow
← Machine Learning and Deep Learning
Lesson 1690 of 3,53836. LLM Inference OptimizationFree lesson

Routing Mechanisms in MoE

Understand how gating networks dynamically route inputs to appropriate experts based on learned token-level routing decisions.

Routing Mechanisms in MoE

What you'll learn: How gating networks intelligently decide which expert(s) should process each input token in a Mixture of Experts architecture.

The Core Problem

In a Mixture of Experts (MoE) model, you have multiple specialized "expert" networks, but you can't afford to run every token through every expert—that would explode your compute costs. Instead, you need a smart router (also called a gating network) that decides which expert(s) should handle each token.

How Routing Works

The gating network is a lightweight learned module—typically just a simple linear layer followed by a softmax—that takes a token's representation as input and outputs a probability distribution over all available experts.

Think of it like a hospital triage nurse: each patient (token) arrives, and the nurse quickly assesses them and routes them to the appropriate specialist (expert) rather than having every doctor examine every patient.

The Routing Decision

For each token independently:

  1. The gating network computes a score for every expert
  2. These scores are converted to probabilities (via softmax)
  3. The top-k experts with highest probabilities are selected (commonly k=1 or k=2)
  4. The token is sent only to those chosen experts
  5. The expert outputs are combined using the gating probabilities as weights

This learned, token-level routing means different tokens can specialize—perhaps syntax-related tokens go to one expert while domain-specific terms go to another.

Why It Matters

Dynamic routing is what makes MoE efficient: you get model capacity from many experts, but only activate a small subset per token, keeping inference costs manageable while maintaining model quality.

Key Takeaway: Gating networks enable sparse activation in MoE by learning to route each token to the most relevant expert(s), balancing computational efficiency with specialization.