Course contentsShow
← Machine Learning and Deep Learning
Lesson 1694 of 3,53836. LLM Inference OptimizationFree lesson

Expert Capacity and Token Dropping

Learn how expert capacity limits tokens per expert and how overflow tokens are handled or dropped during training and inference.

Expert Capacity and Token Dropping

What you'll learn: How MoE models limit the number of tokens each expert can process and what happens when experts become overloaded.

The Core Problem

In Mixture of Experts (MoE) models, the routing mechanism assigns tokens to different experts based on their routing scores. But what happens when too many tokens want to use the same expert? Without limits, one expert might receive thousands of tokens while others sit idle, creating both computational imbalance and memory issues.

Expert Capacity: Setting Hard Limits

Expert capacity is a hard limit on how many tokens a single expert can process in one forward pass. It's typically calculated as:

capacity = (total_tokens / num_experts) × capacity_factor

The capacity_factor (usually 1.0–2.0) provides a buffer. A factor of 1.0 means each expert can handle exactly its "fair share" of tokens. A factor of 1.5 allows 50% overflow to accommodate routing imbalances.

Token Dropping: Handling Overflow

When an expert reaches capacity, additional tokens trying to route to it must be dropped. These overflow tokens:

  • During training: Often skip the expert layer entirely (passing through via residual connections) or get routed to a backup expert
  • During inference: May be processed by their second-choice expert or dropped completely

Think of it like a restaurant with limited seating—once tables are full, new customers either wait (not feasible in parallel processing) or go elsewhere.

This dropping creates a balancing act: too low capacity wastes computation and drops useful tokens; too high capacity causes memory bloat and defeats the efficiency gains of sparsity.

Key Takeaway: Expert capacity limits prevent overloading individual experts, but dropped tokens receive no specialized processing, creating a fundamental trade-off between efficiency and quality that MoE models must carefully balance through routing strategies and capacity factors.