Choosing Clipping Thresholds
What you'll learn: You'll discover practical strategies for selecting the right gradient clipping threshold values to stabilize training without hampering optimization.
The Challenge of "How Much?"
You now know how to clip gradients (by value or by norm), but choosing the actual threshold is more art than science. Set it too high and you won't prevent exploding gradients; set it too low and you'll artificially constrain your model's ability to learn, especially early in training when large updates may be beneficial.
Practical Heuristics
Start with Common Defaults
Most practitioners begin with gradient norm clipping at 1.0 or 5.0. RNNs and transformers often use lower values (0.5–1.0), while CNNs might tolerate higher thresholds (5.0–10.0).
Monitor Gradient Statistics
Track the distribution of gradient norms across training batches. Your threshold should sit above the typical norm (so normal updates pass through) but below the occasional spikes that destabilize training. If you see norms regularly hitting 50+ while most are around 2–3, a threshold of 5.0–10.0 makes sense.
Tune Empirically
Start conservative (e.g., 1.0), train for a few epochs, and check:
- Is training stable but slow? Increase the threshold.
- Are you still seeing loss spikes or NaN values? Decrease it.
Model-Specific Considerations
- RNNs/LSTMs: More prone to exploding gradients; use lower thresholds (0.5–2.0)
- Transformers: Typically 1.0 works well with layer normalization
- Deep ResNets: May need higher thresholds or work fine without clipping
Think of it like setting a speed limit: too slow frustrates everyone, too fast causes crashes. You want the threshold that permits normal "traffic" while preventing dangerous extremes.
Key Takeaway: Gradient clipping thresholds are chosen empirically by monitoring gradient norm distributions—start with defaults like 1.0–5.0, then adjust based on training stability and convergence speed for your specific architecture.