The Sigmoid Function: Properties and Limitations
What you'll learn: You'll understand how the sigmoid activation function works, why its output range makes it useful, and why it causes the vanishing gradient problem.
What is the Sigmoid Function?
The sigmoid function is a smooth, S-shaped curve that transforms any input value into an output between 0 and 1. Mathematically, it's defined as:
σ(z) = 1 / (1 + e^(-z))
Think of sigmoid as a "soft switch" — unlike a hard step function that jumps from 0 to 1, sigmoid gradually transitions, making it differentiable everywhere.
Key Properties
Output Range: Sigmoid always outputs values between 0 and 1, making it naturally interpretable as probabilities. This is why it's commonly used in the output layer for binary classification.
Smooth Gradient: The derivative of sigmoid is σ'(z) = σ(z) × (1 - σ(z)), which is smooth and can be computed efficiently using the function's own output.
Centered Around 0.5: When the input is 0, sigmoid outputs 0.5. Large positive inputs approach 1, while large negative inputs approach 0.
The Vanishing Gradient Problem
Here's the critical limitation: When inputs are very large (positive or negative), the sigmoid function saturates — it becomes nearly flat. In these flat regions, the gradient approaches zero.
During backpropagation, gradients multiply layer by layer. If sigmoid neurons saturate, their tiny gradients get multiplied through the network, shrinking exponentially. This means earlier layers receive almost no learning signal — their weights barely update, and training stalls.
This is especially problematic in deep networks where many multiplications occur.
Key Takeaway: Sigmoid squashes inputs to [0,1] and has a smooth derivative, but it suffers from vanishing gradients when neurons saturate at extreme input values, making it problematic for deep hidden layers.