GLaM and Sparse Gating
What you'll learn: How GLaM uses top-2 routing and sparse gating to enable decoder-only models at trillion-parameter scale.
The Core Innovation
GLaM (Generalist Language Model) from Google demonstrates that you can build decoder-only language models with over a trillion parameters by combining sparse gating with a top-2 routing strategy. Unlike dense models where every token passes through every layer, GLaM activates only a small fraction of its experts per token—making trillion-scale models computationally feasible.
Top-2 Routing Strategy
In GLaM, each token is routed to exactly two experts per MoE layer. The router network computes scores for all available experts, then selects the top 2. These two experts process the token in parallel, and their outputs are combined using the gating scores as weights:
output = gate₁ × expert₁(token) + gate₂ × expert₂(token)
This differs from top-1 (like Switch Transformers) by providing redundancy and richer representations, while still maintaining sparsity—only 2 out of potentially 64 experts activate.
Enabling Trillion-Parameter Scale
The magic is in the math: if you have 64 experts per layer but only activate 2, you pay roughly 1/32 the compute cost of a dense model with equivalent total parameters. GLaM achieves 1.2 trillion parameters while using similar FLOPs to a 175B dense model during inference. The sparse gating mechanism ensures tokens skip most experts entirely, maintaining inference speed despite the massive parameter count.
Key Takeaway: GLaM's top-2 sparse gating proves that decoder-only models can reach trillion-parameter scales efficiently by activating only a tiny fraction of experts per token, balancing model capacity with computational feasibility.