The Feedforward Network: GLU and SwiGLU
What you'll learn: How modern transformer FFN sublayers use gating mechanisms to boost expressiveness beyond simple two-layer networks.
The Traditional FFN Sublayer
In each transformer block, after self-attention, we apply a feedforward network (FFN) to every position independently. The classic design is simple:
FFN(x) = W2 · ReLU(W1 · x + b1) + b2
This expands the dimension (typically 4×), applies non-linearity, then projects back down. While effective, researchers discovered that gating mechanisms can significantly improve what the network learns.
GLU: Gated Linear Units
The Gated Linear Unit (GLU) splits the projection into two paths—one becomes a gate that controls information flow:
GLU(x) = (W1 · x) ⊗ σ(W2 · x)
The ⊗ denotes element-wise multiplication, and σ is typically sigmoid. Think of it like a smart filter: one pathway computes features, while the other decides which features to let through.
SwiGLU: The Modern Standard
SwiGLU combines GLU gating with the Swish activation function (x · sigmoid(x)), creating a powerful variant used in models like PaLM and LLaMA:
SwiGLU(x) = (Swish(W1 · x)) ⊗ (W2 · x)
This design lets the network modulate its own activations adaptively. Instead of fixed non-linearity, the gate learns context-dependent routing of information—making the FFN more expressive without adding depth.
The computational cost increases slightly (extra projection), but empirical results show substantial gains in language modeling performance.
Key Takeaway: Modern transformers replace simple ReLU FFNs with gated variants like SwiGLU, where one pathway gates another, dramatically improving the model's ability to learn complex feature interactions within each layer.