Gradient Norm Monitoring
You'll learn how to track gradient magnitudes during training to catch vanishing or exploding gradients before they derail your model.
What Is Gradient Norm Monitoring?
Gradient norm monitoring means measuring the size (magnitude) of your gradients during training. Think of it like checking your car's temperature gauge while driving—if it's too hot or too cold, something's wrong. Similarly, if gradients are too large (exploding) or too small (vanishing), your network won't train properly.
The "norm" is just a way to measure the overall size of all gradient values together, typically using the L2 norm (Euclidean length), like measuring the diagonal of a box that contains all your gradients.
Why Monitor Gradient Norms?
After learning about vanishing and exploding gradients, you now need practical ways to detect them during training. You can't just wait for training to fail—you want early warning signs.
By logging gradient norms at each training step, you can:
- Spot exploding gradients: Norms suddenly spike to very large values (1e6, 1e10, etc.)
- Catch vanishing gradients: Norms decay toward zero (1e-8, 1e-12, etc.)
- Verify initialization: Check if your Xavier or He initialization is working
- Adjust learning rates: If gradients are consistently large or small, tune accordingly
How to Monitor in Practice
Track the norm of gradients for each layer or the entire network:
- Log gradient norms every few iterations
- Plot them over time to visualize trends
- Set threshold alerts (e.g., warn if norm > 100 or < 1e-6)
Key Takeaway: Gradient norm monitoring is your diagnostic dashboard for training health—regularly checking that gradients are neither exploding nor vanishing lets you intervene early with better initialization, activation functions, or residual connections.