Course contentsShow
← Machine Learning and Deep Learning
Lesson 730 of 3,53817. Optimization for Deep LearningFree lesson

Gradient Clipping in PyTorch

Implementing gradient clipping using torch.nn.utils.clip_grad_norm_ and clip_grad_value_ functions.

Gradient Clipping in PyTorch

What you'll learn: How to implement gradient clipping in PyTorch using built-in utility functions to prevent exploding gradients during training.

The Core Idea

You've learned why and when to clip gradients—now it's time to actually do it. PyTorch provides two convenient functions in torch.nn.utils that handle gradient clipping for you: clip_grad_norm_() and clip_grad_value_(). These functions modify gradients in-place after backpropagation but before the optimizer step.

The Two Clipping Functions

Clipping by Norm

torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm) clips the total gradient norm across all parameters. This is like ensuring the "length" of your gradient vector doesn't exceed a threshold:

loss.backward()  # Compute gradients
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()  # Update weights with clipped gradients

Clipping by Value

torch.nn.utils.clip_grad_value_(model.parameters(), clip_value) clips each gradient element-wise to a range [-clip_value, clip_value]:

loss.backward()
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)
optimizer.step()

Why the Underscore?

The trailing underscore (_) indicates these functions modify gradients in-place—they don't return new values, they change the existing gradients directly.

Typical Training Loop

for batch in dataloader:
    optimizer.zero_grad()
    output = model(batch)
    loss = criterion(output, target)
    loss.backward()
    
    # Clip gradients before optimizer step
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
    
    optimizer.step()

Key Takeaway: PyTorch's clip_grad_norm_() and clip_grad_value_() make gradient clipping a one-line operation that fits naturally between backward() and optimizer.step(), protecting your training from exploding gradients.