Top-K Expert Selection
You'll learn how MoE models use top-k routing to activate only the most relevant experts per token, balancing efficiency and expressiveness.
The Core Idea
In a Mixture of Experts (MoE) model, you don't want to run every token through all experts—that defeats the purpose of sparsity. Instead, you use top-k expert selection: for each input token, a routing mechanism scores all available experts, then activates only the k highest-scoring ones.
Think of it like consulting specialists: when you have a medical question, you don't ask every doctor in the hospital. You pick the top 2-3 specialists whose expertise best matches your specific issue. The routing network acts as a triage system, deciding which experts are most relevant.
How It Works
- Router scores: The routing mechanism (typically a learned linear layer plus softmax) computes a score for each expert given the token's representation
- Select top-k: Choose the k experts with highest scores (commonly k=1 or k=2)
- Weighted combination: Activate only those k experts and combine their outputs, weighted by their normalized router scores
For example, with 8 experts and k=2, each token only passes through 2 experts instead of all 8, reducing computation by ~75% while maintaining model quality.
The Trade-off
- Lower k (e.g., k=1): Maximum efficiency, minimal compute per token, but less expressive
- Higher k (e.g., k=4): More compute-intensive but allows richer token representations through expert combination
- Most production systems use k=2 as a sweet spot
Key Takeaway: Top-k expert selection enables MoE models to maintain sparse computation by routing each token to only k most relevant experts, achieving sub-linear scaling of active parameters relative to total model capacity.