ELU: Exponential Linear Units
What you'll learn: How ELU activation functions use exponential curves for negative inputs to create smoother gradients than ReLU variants.
The Core Idea
ELU (Exponential Linear Unit) takes a different approach to handling negative values compared to ReLU and its variants. Instead of using a straight line (like Leaky ReLU) or learning a slope (like PReLU), ELU uses an exponential function for negative inputs that smoothly approaches a negative saturation value.
The ELU function is defined as:
- For positive values:
f(x) = x(just like ReLU) - For negative values:
f(x) = α(e^x - 1), where α is typically 1.0
Why Exponential Curves?
Think of ELU as a "soft floor" rather than a hard cutoff. When inputs are negative, instead of dropping to zero (ReLU) or using a gentle slope (Leaky ReLU), ELU curves downward exponentially. This creates:
- Smoother gradients: The exponential function is continuously differentiable everywhere, eliminating the sharp corner at zero that ReLU has
- Negative outputs: Like tanh, ELU can produce negative values, which helps push mean activations closer to zero
- Self-normalizing properties: The negative saturation helps control the variance of activations
The Trade-off
The exponential computation (e^x) is more expensive than simple multiplication, making ELU slightly slower than Leaky ReLU or PReLU. However, this computational cost often pays off with faster convergence during training.
Key Takeaway: ELU uses an exponential curve for negative values to achieve smoother gradients and self-normalizing behavior, trading computational cost for improved training dynamics compared to simpler ReLU variants.