Numerical Gradient Checking
What You'll Learn
You'll learn how to verify that your backpropagation implementation is correct by comparing analytical gradients against numerically computed approximations.
The Core Idea
Backpropagation computes gradients using the chain rule—a fast, elegant mathematical approach. But coding it by hand is error-prone: a single sign flip or missing term can silently produce wrong gradients, leading to models that train poorly or not at all.
Numerical gradient checking gives you a way to catch these bugs. It uses the definition of a derivative from calculus:
f'(x) ≈ [f(x + ε) - f(x - ε)] / (2ε)
This finite difference approximation computes the gradient by nudging each parameter slightly in both directions and measuring how the loss changes. It's slow (requires two forward passes per parameter), but it's a simple, independent calculation that doesn't rely on your backpropagation code.
How It Works
- Compute analytical gradients using your backpropagation implementation
- Compute numerical gradients using finite differences for each weight
- Compare them: calculate the relative difference between corresponding gradient values
- If the difference is tiny (typically < 10⁻⁷ for double precision), your backprop is likely correct
Think of It Like This
Imagine checking a complex calculator's multiplication function. You'd test it against doing multiplication by hand for a few examples. Numerical gradient checking is that sanity check for backpropagation—a slow but trustworthy method validating your fast but complex implementation.
When to Use It
Use gradient checking during development and debugging, especially when implementing custom layers or loss functions. Once verified, disable it during actual training—it's too computationally expensive to run every iteration.
Key Takeaway: Numerical gradient checking uses finite differences to approximate gradients, providing an independent verification that your backpropagation implementation computes derivatives correctly.