Course contentsShow
← Machine Learning and Deep Learning
Lesson 726 of 3,53817. Optimization for Deep LearningFree lesson

Gradient Norm and When to Clip

Computing gradient norms and identifying when gradient magnitudes become problematic for optimization.

Gradient Norm and When to Clip

What you'll learn: How to measure gradient magnitude and recognize when it signals optimization trouble.

What Is Gradient Norm?

The gradient norm is a single number that captures the overall magnitude of all gradients in your network. Think of it like measuring wind speed: individual gusts blow in different directions, but the norm tells you the total force.

Mathematically, we typically use the L2 norm (Euclidean length):

||g|| = sqrt(g₁² + g₂² + ... + gₙ²)

where each gᵢ is a gradient for one parameter.

Why Monitor Gradient Norms?

As you learned in the previous lesson about exploding gradients, gradients can grow uncontrollably during backpropagation. The gradient norm gives you a single metric to track this problem:

  • Healthy range: Typically 0.1 to 10
  • Warning signs: Norms consistently above 10-100
  • Critical: Norms in thousands or NaN values

When Should You Clip?

You should consider gradient clipping when:

  1. Training becomes unstable with sudden loss spikes
  2. Gradient norms regularly exceed a threshold (e.g., 5-10)
  3. Working with RNNs or very deep networks where exploding gradients are common
  4. Parameters receive wildly different update magnitudes

Think of clipping like a speed governor on a vehicle—it prevents dangerous acceleration while allowing normal operation.

Key Takeaway: Gradient norm measures total gradient magnitude across all parameters; values consistently above 10 signal exploding gradients and indicate you should apply gradient clipping to stabilize training.