Course contentsShow
← Machine Learning and Deep Learning
Lesson 1696 of 3,53836. LLM Inference OptimizationFree lesson

Switch Transformers Architecture

Study Google's Switch Transformer design with simplified single-expert routing and massive expert counts for extreme scaling.

Switch Transformers Architecture

What you'll learn: How Google's Switch Transformer simplifies Mixture of Experts by routing each token to exactly one expert, enabling unprecedented scaling to trillions of parameters.

The Core Innovation

After exploring MoE routing mechanisms and load balancing challenges, Switch Transformers introduce a radical simplification: route each token to only one expert (instead of top-2 or top-k). This "switch" routing removes complexity while enabling massive expert counts.

Think of it like a train station: instead of each passenger boarding multiple trains (top-k routing), each passenger boards exactly one train. The routing decision is simpler, faster, and the station can handle many more platforms (experts).

Key Design Choices

Simplified Routing

The gating function uses softmax over all experts, then selects the single expert with highest probability:

expert_id = argmax(softmax(router_weights · token_embedding))

No weighted combination—just a hard switch. This reduces computation and memory overhead dramatically.

Massive Expert Scaling

By simplifying routing, Switch Transformers scaled to 2048 experts in some layers. Compare this to earlier MoE models with 8-16 experts. More experts means more specialization and parameter efficiency.

Selective Expert Placement

Not every layer needs MoE—Switch Transformers typically replace only the feedforward layers with expert layers, keeping attention layers dense.

Load Balancing Refinements

With single-expert routing, load balancing becomes critical. Switch Transformers use auxiliary losses and expert capacity management, but simpler routing makes balancing easier to reason about.

Why It Matters for Inference

Single-expert routing means each token activates fewer parameters during inference, maintaining speed despite enormous total parameter counts. A 1.6 trillion parameter Switch Transformer can match smaller dense models' speed.

Key Takeaway: Switch Transformers achieve extreme scaling by simplifying MoE to single-expert routing, enabling thousands of experts while keeping inference costs manageable through sparse activation.