Gradient Norm and When to Clip
What you'll learn: How to measure gradient magnitude and recognize when it signals optimization trouble.
What Is Gradient Norm?
The gradient norm is a single number that captures the overall magnitude of all gradients in your network. Think of it like measuring wind speed: individual gusts blow in different directions, but the norm tells you the total force.
Mathematically, we typically use the L2 norm (Euclidean length):
||g|| = sqrt(g₁² + g₂² + ... + gₙ²)
where each gᵢ is a gradient for one parameter.
Why Monitor Gradient Norms?
As you learned in the previous lesson about exploding gradients, gradients can grow uncontrollably during backpropagation. The gradient norm gives you a single metric to track this problem:
- Healthy range: Typically 0.1 to 10
- Warning signs: Norms consistently above 10-100
- Critical: Norms in thousands or NaN values
When Should You Clip?
You should consider gradient clipping when:
- Training becomes unstable with sudden loss spikes
- Gradient norms regularly exceed a threshold (e.g., 5-10)
- Working with RNNs or very deep networks where exploding gradients are common
- Parameters receive wildly different update magnitudes
Think of clipping like a speed governor on a vehicle—it prevents dangerous acceleration while allowing normal operation.
Key Takeaway: Gradient norm measures total gradient magnitude across all parameters; values consistently above 10 signal exploding gradients and indicate you should apply gradient clipping to stabilize training.