Course contentsShow
← Machine Learning and Deep Learning
Lesson 1692 of 3,53836. LLM Inference OptimizationFree lesson

Top-K Expert Selection

Learn how top-k routing selects a subset of experts per token, balancing compute efficiency with model expressiveness.

Top-K Expert Selection

You'll learn how MoE models use top-k routing to activate only the most relevant experts per token, balancing efficiency and expressiveness.

The Core Idea

In a Mixture of Experts (MoE) model, you don't want to run every token through all experts—that defeats the purpose of sparsity. Instead, you use top-k expert selection: for each input token, a routing mechanism scores all available experts, then activates only the k highest-scoring ones.

Think of it like consulting specialists: when you have a medical question, you don't ask every doctor in the hospital. You pick the top 2-3 specialists whose expertise best matches your specific issue. The routing network acts as a triage system, deciding which experts are most relevant.

How It Works

  1. Router scores: The routing mechanism (typically a learned linear layer plus softmax) computes a score for each expert given the token's representation
  2. Select top-k: Choose the k experts with highest scores (commonly k=1 or k=2)
  3. Weighted combination: Activate only those k experts and combine their outputs, weighted by their normalized router scores

For example, with 8 experts and k=2, each token only passes through 2 experts instead of all 8, reducing computation by ~75% while maintaining model quality.

The Trade-off

  • Lower k (e.g., k=1): Maximum efficiency, minimal compute per token, but less expressive
  • Higher k (e.g., k=4): More compute-intensive but allows richer token representations through expert combination
  • Most production systems use k=2 as a sweet spot

Key Takeaway: Top-k expert selection enables MoE models to maintain sparse computation by routing each token to only k most relevant experts, achieving sub-linear scaling of active parameters relative to total model capacity.