Higher-Order Derivatives
You'll learn how to compute second derivatives and beyond by instructing PyTorch to build a computational graph of the backward pass itself.
The Core Idea
So far, when you call .backward() on a loss, PyTorch computes first derivatives (gradients) of that loss with respect to your parameters. But what if you need the derivative of a derivative—a second-order gradient?
This comes up in certain optimization algorithms, physics-informed neural networks, or when analyzing loss surface curvature. By default, PyTorch's computational graph is discarded after .backward(). To compute higher-order derivatives, you need to preserve that graph so you can differentiate through the gradient computation itself.
How It Works
The magic parameter is create_graph=True in the .backward() call:
import torch
# Function: f(x) = x^3
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
# First derivative (dy/dx)
y.backward(create_graph=True) # Keep the graph!
first_deriv = x.grad.clone() # dy/dx = 3x^2 = 12
# Second derivative (d²y/dx²)
x.grad.zero_()
first_deriv.backward() # Differentiate the gradient
second_deriv = x.grad # d²y/dx² = 6x = 12
Without create_graph=True, the first .backward() would destroy the graph, making the second derivative computation impossible.
Analogy
Think of it like filming a magician's trick. The first backward pass is watching the trick happen. With create_graph=True, you're recording it—so you can rewind and analyze how the trick was performed, enabling deeper inspection.
Key Takeaway:
Use create_graph=True in .backward() to preserve the computational graph and enable higher-order derivative computations by differentiating through the gradient calculation itself.