Why Raw Gradients Are Noisy
You'll learn why vanilla gradient-based saliency maps produce unreliable, noisy explanations despite being mathematically straightforward.
The Core Problem
When you compute raw gradients of a network's output with respect to input pixels (the vanilla gradient method), you're asking: "How would changing this input slightly affect the output?" In theory, large gradients should highlight important features. In practice, you often get what looks like random noise.
Why the Noise Happens
Saturation effects: Neural networks use activation functions like ReLU or sigmoid. When neurons saturate (flatten out), their gradients become zero or near-zero, even if those features are genuinely important. The gradient "dies" not because the feature doesn't matter, but because the neuron has hit its flat region.
Network architecture sensitivity: The gradient signal must backpropagate through many layers. Each layer's weights, biases, and activation choices distort this signal. Deep networks especially suffer—by the time gradients reach the input, they've been transformed through dozens of nonlinear operations, accumulating numerical artifacts.
Local linearity assumption: Gradients assume your model is locally linear around the input. But neural networks are highly nonlinear. A tiny perturbation might hit a decision boundary, causing gradients to point in misleading directions.
Think of it like listening to a whisper after it's echoed through a canyon—the original message gets distorted by all the reflecting surfaces (layers), sometimes completely garbled.
Practical Consequences
You'll see saliency maps with high-frequency noise, highlighting seemingly random pixels rather than coherent objects. Two similar images may produce wildly different gradient patterns. This unreliability is why researchers developed refined methods like Integrated Gradients and SmoothGrad.
Key Takeaway: Raw gradients are noisy because neural networks saturate, architectures introduce distortions during backpropagation, and the local linearity assumption breaks down—requiring more sophisticated attribution methods for reliable interpretability.