Course contentsShow
← Machine Learning and Deep Learning
Lesson 1700 of 3,53836. LLM Inference OptimizationFree lesson

Fine-Grained vs Coarse-Grained MoE

Compare token-level expert routing versus layer-level or block-level MoE designs and their tradeoffs for deployment.

Fine-Grained vs Coarse-Grained MoE

You'll learn: How the granularity of expert routing—token-level versus layer/block-level—affects MoE deployment complexity and performance.

The Granularity Question

In Mixture of Experts models, you must decide when and how often to route through experts. This granularity choice fundamentally shapes your system's behavior and deployment costs.

Token-Level (Fine-Grained) MoE

Fine-grained MoE routes every token independently through experts at each MoE layer. Think of it like a restaurant where each customer (token) chooses their own chef (expert) for every course.

Advantages:

  • Maximum flexibility: each token gets specialized processing
  • Better model quality potential through precise routing
  • Aligns with the original MoE vision (Switch Transformers, Mixtral)

Deployment challenges:

  • Extreme load imbalance: some experts may be idle while others are overloaded
  • Higher memory bandwidth requirements from KV cache fragmentation
  • Complex scheduling and batching across heterogeneous expert selections

Layer-Level or Block-Level (Coarse-Grained) MoE

Coarse-grained MoE makes routing decisions less frequently—perhaps routing entire sequences to the same experts for multiple layers, or activating expert subsets per batch rather than per token.

Advantages:

  • Simplified deployment: predictable memory and compute patterns
  • Better hardware utilization with larger, uniform workloads
  • Easier to implement prefix caching and KV cache optimization

Trade-offs:

  • Reduced routing flexibility may limit model expressiveness
  • Less dynamic adaptation to diverse token needs

Key Takeaway: Fine-grained (token-level) MoE maximizes routing precision but complicates deployment with unpredictable load patterns, while coarse-grained (layer/block-level) MoE sacrifices some flexibility for simpler, more efficient inference infrastructure.