Course contentsShow
← Machine Learning and Deep Learning
Lesson 631 of 3,53815. Neural Network FoundationsFree lesson

Gradient of Common Activation Functions

Deriving analytical gradients for sigmoid, tanh, and ReLU activations used in backpropagation.

Gradient of Common Activation Functions

What you'll learn: How to compute the derivatives of sigmoid, tanh, and ReLU activation functions—the building blocks needed for backpropagation.

Why Activation Gradients Matter

During backpropagation, the chain rule requires us to multiply gradients as we move backward through layers. At each neuron, we need the gradient of the activation function applied to that neuron's weighted sum. Without these derivatives, we can't update weights efficiently.

The Three Core Activation Gradients

Sigmoid: σ(z) = 1/(1 + e^(-z))

The sigmoid's gradient has an elegant property: σ'(z) = σ(z) · (1 - σ(z))

This means if you've already computed the forward pass output, you can reuse it! The gradient is largest around z=0 (≈0.25) and vanishes as |z| grows large—a problem called "vanishing gradients."

Tanh: tanh(z) = (e^z - e^(-z))/(e^z + e^(-z))

The tanh gradient is similarly convenient: tanh'(z) = 1 - tanh²(z)

Again, you can reuse the forward activation value. Tanh gradients are stronger than sigmoid (max ≈1.0 at z=0) but still vanish for extreme inputs.

ReLU: ReLU(z) = max(0, z)

ReLU has the simplest gradient: ReLU'(z) = 1 if z > 0, else 0

It's a step function—either the gradient flows fully (1) or dies completely (0). This simplicity makes ReLU computationally cheap and helps avoid vanishing gradients for positive values, though neurons can "die" when z ≤ 0.

Practical Impact

These formulas directly plug into the chain rule during backpropagation. When computing gradients for layer l, you multiply the upstream gradient by the local activation gradient, enabling weight updates throughout the network.

Key Takeaway: The gradients of sigmoid, tanh, and ReLU are simple formulas that enable efficient backpropagation—sigmoid and tanh can reuse forward pass values, while ReLU's binary gradient makes it the fastest to compute.