Gradient of Common Activation Functions
What you'll learn: How to compute the derivatives of sigmoid, tanh, and ReLU activation functions—the building blocks needed for backpropagation.
Why Activation Gradients Matter
During backpropagation, the chain rule requires us to multiply gradients as we move backward through layers. At each neuron, we need the gradient of the activation function applied to that neuron's weighted sum. Without these derivatives, we can't update weights efficiently.
The Three Core Activation Gradients
Sigmoid: σ(z) = 1/(1 + e^(-z))
The sigmoid's gradient has an elegant property: σ'(z) = σ(z) · (1 - σ(z))
This means if you've already computed the forward pass output, you can reuse it! The gradient is largest around z=0 (≈0.25) and vanishes as |z| grows large—a problem called "vanishing gradients."
Tanh: tanh(z) = (e^z - e^(-z))/(e^z + e^(-z))
The tanh gradient is similarly convenient: tanh'(z) = 1 - tanh²(z)
Again, you can reuse the forward activation value. Tanh gradients are stronger than sigmoid (max ≈1.0 at z=0) but still vanish for extreme inputs.
ReLU: ReLU(z) = max(0, z)
ReLU has the simplest gradient: ReLU'(z) = 1 if z > 0, else 0
It's a step function—either the gradient flows fully (1) or dies completely (0). This simplicity makes ReLU computationally cheap and helps avoid vanishing gradients for positive values, though neurons can "die" when z ≤ 0.
Practical Impact
These formulas directly plug into the chain rule during backpropagation. When computing gradients for layer l, you multiply the upstream gradient by the local activation gradient, enabling weight updates throughout the network.
Key Takeaway: The gradients of sigmoid, tanh, and ReLU are simple formulas that enable efficient backpropagation—sigmoid and tanh can reuse forward pass values, while ReLU's binary gradient makes it the fastest to compute.