GELU: Gaussian Error Linear Units
You'll learn what GELU is and why it's become the go-to activation function in modern transformer architectures.
What is GELU?
GELU (Gaussian Error Linear Unit) is a smooth, stochastic-inspired activation function that multiplies the input by its probability of being above zero under a Gaussian distribution. Unlike the sharp corners of ReLU or Leaky ReLU, GELU provides a probabilistic, curved approach to activation.
The Core Idea
Think of GELU as asking: "How confident am I that this input should be activated?" Rather than a hard cutoff (like ReLU's zero threshold) or a fixed slope (like Leaky ReLU), GELU weights each input by the likelihood it comes from the positive side of a normal distribution.
The mathematical formulation is:
GELU(x) = x * Φ(x)
Where Φ(x) is the cumulative distribution function (CDF) of the standard Gaussian distribution. In plain terms: you multiply the input by the probability that a Gaussian random variable is less than or equal to that input.
Why GELU Matters
GELU gained prominence in transformer models (like BERT and GPT) because:
- Smoothness: Unlike ReLU's sharp corner at zero, GELU is differentiable everywhere, which can improve gradient flow
- Stochastic regularization: The probabilistic weighting acts as implicit regularization
- Empirical performance: It consistently outperforms ReLU and ELU in large-scale language models
For negative inputs, GELU doesn't cut off completely to zero—it gently dampens them, similar to ELU but with a different mathematical foundation.
Key Takeaway: GELU is a smooth, probabilistically-motivated activation function that weights inputs by their Gaussian likelihood, making it ideal for transformer architectures and modern deep learning models where smoothness and sophisticated gradient behavior matter.