Course contentsShow
← Machine Learning and Deep Learning
Lesson 729 of 3,53817. Optimization for Deep LearningFree lesson

Choosing Clipping Thresholds

Heuristics and empirical strategies for selecting appropriate gradient clipping values for your model.

Choosing Clipping Thresholds

What you'll learn: You'll discover practical strategies for selecting the right gradient clipping threshold values to stabilize training without hampering optimization.

The Challenge of "How Much?"

You now know how to clip gradients (by value or by norm), but choosing the actual threshold is more art than science. Set it too high and you won't prevent exploding gradients; set it too low and you'll artificially constrain your model's ability to learn, especially early in training when large updates may be beneficial.

Practical Heuristics

Start with Common Defaults

Most practitioners begin with gradient norm clipping at 1.0 or 5.0. RNNs and transformers often use lower values (0.5–1.0), while CNNs might tolerate higher thresholds (5.0–10.0).

Monitor Gradient Statistics

Track the distribution of gradient norms across training batches. Your threshold should sit above the typical norm (so normal updates pass through) but below the occasional spikes that destabilize training. If you see norms regularly hitting 50+ while most are around 2–3, a threshold of 5.0–10.0 makes sense.

Tune Empirically

Start conservative (e.g., 1.0), train for a few epochs, and check:

  • Is training stable but slow? Increase the threshold.
  • Are you still seeing loss spikes or NaN values? Decrease it.

Model-Specific Considerations

  • RNNs/LSTMs: More prone to exploding gradients; use lower thresholds (0.5–2.0)
  • Transformers: Typically 1.0 works well with layer normalization
  • Deep ResNets: May need higher thresholds or work fine without clipping

Think of it like setting a speed limit: too slow frustrates everyone, too fast causes crashes. You want the threshold that permits normal "traffic" while preventing dangerous extremes.

Key Takeaway: Gradient clipping thresholds are chosen empirically by monitoring gradient norm distributions—start with defaults like 1.0–5.0, then adjust based on training stability and convergence speed for your specific architecture.