Course contentsShow
← Machine Learning and Deep Learning
Lesson 1609 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

The Feedforward Network: GLU and SwiGLU

Explore the role of the FFN sublayer and modern gated variants like SwiGLU that improve expressiveness.

The Feedforward Network: GLU and SwiGLU

What you'll learn: How modern transformer FFN sublayers use gating mechanisms to boost expressiveness beyond simple two-layer networks.

The Traditional FFN Sublayer

In each transformer block, after self-attention, we apply a feedforward network (FFN) to every position independently. The classic design is simple:

FFN(x) = W2 · ReLU(W1 · x + b1) + b2

This expands the dimension (typically 4×), applies non-linearity, then projects back down. While effective, researchers discovered that gating mechanisms can significantly improve what the network learns.

GLU: Gated Linear Units

The Gated Linear Unit (GLU) splits the projection into two paths—one becomes a gate that controls information flow:

GLU(x) = (W1 · x) ⊗ σ(W2 · x)

The ⊗ denotes element-wise multiplication, and σ is typically sigmoid. Think of it like a smart filter: one pathway computes features, while the other decides which features to let through.

SwiGLU: The Modern Standard

SwiGLU combines GLU gating with the Swish activation function (x · sigmoid(x)), creating a powerful variant used in models like PaLM and LLaMA:

SwiGLU(x) = (Swish(W1 · x)) ⊗ (W2 · x)

This design lets the network modulate its own activations adaptively. Instead of fixed non-linearity, the gate learns context-dependent routing of information—making the FFN more expressive without adding depth.

The computational cost increases slightly (extra projection), but empirical results show substantial gains in language modeling performance.

Key Takeaway: Modern transformers replace simple ReLU FFNs with gated variants like SwiGLU, where one pathway gates another, dramatically improving the model's ability to learn complex feature interactions within each layer.