Course contentsShow
← Machine Learning and Deep Learning
Lesson 658 of 3,53816. Activation Functions and Weight InitializationFree lesson

ELU: Exponential Linear Units

Using exponential functions for negative values to achieve smoother gradients and outputs.

ELU: Exponential Linear Units

What you'll learn: How ELU activation functions use exponential curves for negative inputs to create smoother gradients than ReLU variants.

The Core Idea

ELU (Exponential Linear Unit) takes a different approach to handling negative values compared to ReLU and its variants. Instead of using a straight line (like Leaky ReLU) or learning a slope (like PReLU), ELU uses an exponential function for negative inputs that smoothly approaches a negative saturation value.

The ELU function is defined as:

  • For positive values: f(x) = x (just like ReLU)
  • For negative values: f(x) = α(e^x - 1), where α is typically 1.0

Why Exponential Curves?

Think of ELU as a "soft floor" rather than a hard cutoff. When inputs are negative, instead of dropping to zero (ReLU) or using a gentle slope (Leaky ReLU), ELU curves downward exponentially. This creates:

  1. Smoother gradients: The exponential function is continuously differentiable everywhere, eliminating the sharp corner at zero that ReLU has
  2. Negative outputs: Like tanh, ELU can produce negative values, which helps push mean activations closer to zero
  3. Self-normalizing properties: The negative saturation helps control the variance of activations

The Trade-off

The exponential computation (e^x) is more expensive than simple multiplication, making ELU slightly slower than Leaky ReLU or PReLU. However, this computational cost often pays off with faster convergence during training.

Key Takeaway: ELU uses an exponential curve for negative values to achieve smoother gradients and self-normalizing behavior, trading computational cost for improved training dynamics compared to simpler ReLU variants.