Hyperbolic Tangent (tanh): Zero-Centered Activation
What You'll Learn
You'll discover how the tanh activation function addresses a key limitation of sigmoid by producing zero-centered outputs, making gradient descent more efficient.
The Problem with Sigmoid
Recall that the sigmoid function outputs values between 0 and 1, meaning all activations are positive. This creates a bias in gradient updates during backpropagation, potentially slowing down learning. The network struggles when all inputs to a neuron have the same sign.
What is tanh?
The hyperbolic tangent function is mathematically defined as:
tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))
Unlike sigmoid, tanh outputs values between -1 and +1, with the crucial property that tanh(0) = 0. This makes it zero-centered.
Why Zero-Centering Matters
Think of learning as navigating a landscape. When all gradient components point in the same direction (sigmoid's problem), you can only move in certain diagonal directions, requiring zigzagging. Zero-centered outputs from tanh allow gradients to be both positive and negative, enabling more direct paths during optimization.
Gradient Properties
The derivative of tanh is:
tanh'(x) = 1 - tanh²(x)
This peaks at 1 when x=0 (better than sigmoid's maximum of 0.25) but still suffers from vanishing gradients for large |x| values, approaching zero at the extremes.
Practical Advantage
tanh typically leads to faster convergence than sigmoid in hidden layers because the zero-centered outputs prevent systematic bias in weight updates across the network.
Key Takeaway: tanh improves upon sigmoid by outputting values in [-1, 1] with zero-centered activations, enabling more efficient gradient descent, though it still faces vanishing gradient issues for extreme input values.