Course contentsShow
← Machine Learning and Deep Learning
Lesson 653 of 3,53816. Activation Functions and Weight InitializationFree lesson

Hyperbolic Tangent (tanh): Zero-Centered Activation

How tanh improves on sigmoid with zero-centered outputs and its gradient properties.

Hyperbolic Tangent (tanh): Zero-Centered Activation

What You'll Learn

You'll discover how the tanh activation function addresses a key limitation of sigmoid by producing zero-centered outputs, making gradient descent more efficient.

The Problem with Sigmoid

Recall that the sigmoid function outputs values between 0 and 1, meaning all activations are positive. This creates a bias in gradient updates during backpropagation, potentially slowing down learning. The network struggles when all inputs to a neuron have the same sign.

What is tanh?

The hyperbolic tangent function is mathematically defined as:

tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))

Unlike sigmoid, tanh outputs values between -1 and +1, with the crucial property that tanh(0) = 0. This makes it zero-centered.

Why Zero-Centering Matters

Think of learning as navigating a landscape. When all gradient components point in the same direction (sigmoid's problem), you can only move in certain diagonal directions, requiring zigzagging. Zero-centered outputs from tanh allow gradients to be both positive and negative, enabling more direct paths during optimization.

Gradient Properties

The derivative of tanh is:

tanh'(x) = 1 - tanh²(x)

This peaks at 1 when x=0 (better than sigmoid's maximum of 0.25) but still suffers from vanishing gradients for large |x| values, approaching zero at the extremes.

Practical Advantage

tanh typically leads to faster convergence than sigmoid in hidden layers because the zero-centered outputs prevent systematic bias in weight updates across the network.

Key Takeaway: tanh improves upon sigmoid by outputting values in [-1, 1] with zero-centered activations, enabling more efficient gradient descent, though it still faces vanishing gradient issues for extreme input values.