Course contentsShow
← Machine Learning and Deep Learning
Lesson 1697 of 3,53836. LLM Inference OptimizationFree lesson

GLaM and Sparse Gating

Examine GLaM's top-2 routing strategy and how sparse gating enables decoder-only models with trillion-parameter scales.

GLaM and Sparse Gating

What you'll learn: How GLaM uses top-2 routing and sparse gating to enable decoder-only models at trillion-parameter scale.

The Core Innovation

GLaM (Generalist Language Model) from Google demonstrates that you can build decoder-only language models with over a trillion parameters by combining sparse gating with a top-2 routing strategy. Unlike dense models where every token passes through every layer, GLaM activates only a small fraction of its experts per token—making trillion-scale models computationally feasible.

Top-2 Routing Strategy

In GLaM, each token is routed to exactly two experts per MoE layer. The router network computes scores for all available experts, then selects the top 2. These two experts process the token in parallel, and their outputs are combined using the gating scores as weights:

output = gate₁ × expert₁(token) + gate₂ × expert₂(token)

This differs from top-1 (like Switch Transformers) by providing redundancy and richer representations, while still maintaining sparsity—only 2 out of potentially 64 experts activate.

Enabling Trillion-Parameter Scale

The magic is in the math: if you have 64 experts per layer but only activate 2, you pay roughly 1/32 the compute cost of a dense model with equivalent total parameters. GLaM achieves 1.2 trillion parameters while using similar FLOPs to a 175B dense model during inference. The sparse gating mechanism ensures tokens skip most experts entirely, maintaining inference speed despite the massive parameter count.

Key Takeaway: GLaM's top-2 sparse gating proves that decoder-only models can reach trillion-parameter scales efficiently by activating only a tiny fraction of experts per token, balancing model capacity with computational feasibility.