Course contentsShow
← Machine Learning and Deep Learning
Lesson 680 of 3,53816. Activation Functions and Weight InitializationFree lesson

Gradient Norm Monitoring

Practical techniques for tracking gradient magnitudes during training to diagnose flow problems early.

Gradient Norm Monitoring

You'll learn how to track gradient magnitudes during training to catch vanishing or exploding gradients before they derail your model.

What Is Gradient Norm Monitoring?

Gradient norm monitoring means measuring the size (magnitude) of your gradients during training. Think of it like checking your car's temperature gauge while driving—if it's too hot or too cold, something's wrong. Similarly, if gradients are too large (exploding) or too small (vanishing), your network won't train properly.

The "norm" is just a way to measure the overall size of all gradient values together, typically using the L2 norm (Euclidean length), like measuring the diagonal of a box that contains all your gradients.

Why Monitor Gradient Norms?

After learning about vanishing and exploding gradients, you now need practical ways to detect them during training. You can't just wait for training to fail—you want early warning signs.

By logging gradient norms at each training step, you can:

  • Spot exploding gradients: Norms suddenly spike to very large values (1e6, 1e10, etc.)
  • Catch vanishing gradients: Norms decay toward zero (1e-8, 1e-12, etc.)
  • Verify initialization: Check if your Xavier or He initialization is working
  • Adjust learning rates: If gradients are consistently large or small, tune accordingly

How to Monitor in Practice

Track the norm of gradients for each layer or the entire network:

  • Log gradient norms every few iterations
  • Plot them over time to visualize trends
  • Set threshold alerts (e.g., warn if norm > 100 or < 1e-6)

Key Takeaway: Gradient norm monitoring is your diagnostic dashboard for training health—regularly checking that gradients are neither exploding nor vanishing lets you intervene early with better initialization, activation functions, or residual connections.