Gradient Clipping in PyTorch
What you'll learn: How to implement gradient clipping in PyTorch using built-in utility functions to prevent exploding gradients during training.
The Core Idea
You've learned why and when to clip gradients—now it's time to actually do it. PyTorch provides two convenient functions in torch.nn.utils that handle gradient clipping for you: clip_grad_norm_() and clip_grad_value_(). These functions modify gradients in-place after backpropagation but before the optimizer step.
The Two Clipping Functions
Clipping by Norm
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) clips the total gradient norm across all parameters. This is like ensuring the "length" of your gradient vector doesn't exceed a threshold:
loss.backward() # Compute gradients
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step() # Update weights with clipped gradients
Clipping by Value
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value) clips each gradient element-wise to a range [-clip_value, clip_value]:
loss.backward()
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)
optimizer.step()
Why the Underscore?
The trailing underscore (_) indicates these functions modify gradients in-place—they don't return new values, they change the existing gradients directly.
Typical Training Loop
for batch in dataloader:
optimizer.zero_grad()
output = model(batch)
loss = criterion(output, target)
loss.backward()
# Clip gradients before optimizer step
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
Key Takeaway: PyTorch's clip_grad_norm_() and clip_grad_value_() make gradient clipping a one-line operation that fits naturally between backward() and optimizer.step(), protecting your training from exploding gradients.