Course contentsShow
← Machine Learning and Deep Learning
Lesson 727 of 3,53817. Optimization for Deep LearningFree lesson

Gradient Clipping by Value

Clipping individual gradient components to a fixed range to prevent extreme parameter updates.

Gradient Clipping by Value

What you'll learn: How to cap individual gradient components within a fixed range to prevent any single parameter from receiving an extreme update.

The Core Idea

You've learned about the exploding gradient problem and gradient norm clipping. Gradient clipping by value takes a different approach: instead of scaling the entire gradient vector, it clips each individual gradient component independently to stay within a specified range, typically [-threshold, +threshold].

Think of it like putting guardrails on a winding mountain road. While gradient norm clipping shrinks the entire direction vector proportionally (like slowing down the whole car), clipping by value prevents any individual dimension from going too far off course (like having separate barriers on each side of each lane).

How It Works

For each gradient component g:

if g > threshold:
    g = threshold
elif g < -threshold:
    g = -threshold

If you have a gradient vector [0.5, 10.3, -8.2, 0.1] and clip at ±5, you get [0.5, 5.0, -5.0, 0.1]. Notice only the extreme values changed—the others remain untouched.

When to Use It

Gradient clipping by value is simpler and more localized than norm clipping. It's useful when you want to prevent individual parameters from making wild jumps, but it doesn't preserve the gradient's original direction like norm clipping does. You've already seen norm clipping is more common in practice, but value clipping can be effective for certain architectures or when you want parameter-specific control.

Key Takeaway: Gradient clipping by value caps each gradient component independently to a fixed range, preventing extreme updates to individual parameters without changing the relative magnitudes of small gradients.