Course contentsShow
← Machine Learning and Deep Learning
Lesson 659 of 3,53816. Activation Functions and Weight InitializationFree lesson

GELU: Gaussian Error Linear Units

The stochastic activation function used in transformers and modern architectures.

GELU: Gaussian Error Linear Units

You'll learn what GELU is and why it's become the go-to activation function in modern transformer architectures.

What is GELU?

GELU (Gaussian Error Linear Unit) is a smooth, stochastic-inspired activation function that multiplies the input by its probability of being above zero under a Gaussian distribution. Unlike the sharp corners of ReLU or Leaky ReLU, GELU provides a probabilistic, curved approach to activation.

The Core Idea

Think of GELU as asking: "How confident am I that this input should be activated?" Rather than a hard cutoff (like ReLU's zero threshold) or a fixed slope (like Leaky ReLU), GELU weights each input by the likelihood it comes from the positive side of a normal distribution.

The mathematical formulation is:

GELU(x) = x * Φ(x)

Where Φ(x) is the cumulative distribution function (CDF) of the standard Gaussian distribution. In plain terms: you multiply the input by the probability that a Gaussian random variable is less than or equal to that input.

Why GELU Matters

GELU gained prominence in transformer models (like BERT and GPT) because:

  • Smoothness: Unlike ReLU's sharp corner at zero, GELU is differentiable everywhere, which can improve gradient flow
  • Stochastic regularization: The probabilistic weighting acts as implicit regularization
  • Empirical performance: It consistently outperforms ReLU and ELU in large-scale language models

For negative inputs, GELU doesn't cut off completely to zero—it gently dampens them, similar to ELU but with a different mathematical foundation.

Key Takeaway: GELU is a smooth, probabilistically-motivated activation function that weights inputs by their Gaussian likelihood, making it ideal for transformer architectures and modern deep learning models where smoothness and sophisticated gradient behavior matter.