Course contentsShow
← Machine Learning and Deep Learning
Lesson 728 of 3,53817. Optimization for Deep LearningFree lesson

Gradient Clipping by Norm

Rescaling the entire gradient vector when its L2 norm exceeds a threshold while preserving direction.

Gradient Clipping by Norm

What you'll learn: How to prevent exploding gradients by rescaling the entire gradient vector when it becomes too large, while keeping its direction intact.


The Core Idea

Gradient clipping by norm (also called global norm clipping) works differently than clipping by value. Instead of capping individual gradient components independently, you look at the overall magnitude of the entire gradient vector using its L2 norm (Euclidean length). If this norm exceeds your chosen threshold, you scale down the entire gradient proportionally so that its norm equals the threshold.

Think of it like this: imagine you're hiking and your GPS says to move 100 meters northeast. That's too far, so you decide your maximum step is 10 meters. Instead of walking 10 meters north and 10 meters east (changing direction), you walk 10 meters in the exact same northeast direction — just shorter.

The Mathematics

Given a gradient vector g and a threshold max_norm:

  1. Calculate the L2 norm: ||g|| = sqrt(sum of all squared gradient values)
  2. If ||g|| > max_norm:
    • Compute the scaling factor: scale = max_norm / ||g||
    • Rescale: g_clipped = scale × g

This preserves the gradient's direction but limits its magnitude.

Why This Matters

Clipping by norm is generally superior to clipping by value because it maintains the relative relationships between all gradient components. You're still pointing the optimizer in the right direction — you've just limited how big a step it can take. This is especially important in RNNs where gradients from many time steps combine.

Key Takeaway: Gradient clipping by norm rescales the entire gradient vector proportionally when its magnitude exceeds a threshold, preserving optimization direction while preventing explosive updates.