Gradient Saturation and Dead Neurons
What you'll learn: Why saliency methods sometimes fail to highlight important features due to zero gradients from saturated activations and dead neurons.
The Problem with Zero Gradients
Saliency methods like Vanilla Gradient, SmoothGrad, and Gradient × Input rely on backpropagating gradients to identify which input features matter most. But what happens when gradients become zero? The saliency map goes blank, even if those features are critically important.
Two Common Causes
Gradient Saturation
When activation functions like sigmoid or tanh receive very large or very small inputs, they produce outputs near their asymptotes (0/1 for sigmoid, -1/1 for tanh). In these "saturated" regions, the derivative is nearly zero. During backpropagation, these tiny gradients multiply through the chain rule, producing negligible or zero saliency values for the corresponding input features.
Think of it like a flattened hill: no matter which direction you nudge the input, the output barely changes, so gradient-based methods conclude (incorrectly) that the feature doesn't matter.
Dead ReLUs
ReLU neurons output zero for all negative inputs, and their gradient is exactly zero in that region. If a neuron's weights shift such that it always receives negative inputs, it becomes "dead"—permanently outputting zero with zero gradient. Any input feature flowing through dead neurons will show zero saliency, even if switching that feature's value would reactivate the neuron and dramatically change the prediction.
Why This Matters for Interpretability
Gradient-based saliency maps can miss important features entirely when neurons saturate or die. The model may heavily rely on a feature, but if the gradient path is blocked by saturation, your saliency map won't reveal it. This is why methods like GradCAM, Guided Backpropagation, and ensemble approaches exist—they attempt to work around these gradient pathologies.
Key Takeaway: Saturated activations and dead ReLUs produce zero gradients that cause gradient-based saliency methods to overlook genuinely important features, highlighting a fundamental limitation of these interpretability techniques.