Course contentsShow
← Machine Learning and Deep Learning
Lesson 799 of 3,53819. PyTorch FundamentalsFree lesson

Higher-Order Derivatives

Computing second derivatives and higher-order gradients using create_graph=True in backward().

Higher-Order Derivatives

You'll learn how to compute second derivatives and beyond by instructing PyTorch to build a computational graph of the backward pass itself.

The Core Idea

So far, when you call .backward() on a loss, PyTorch computes first derivatives (gradients) of that loss with respect to your parameters. But what if you need the derivative of a derivative—a second-order gradient?

This comes up in certain optimization algorithms, physics-informed neural networks, or when analyzing loss surface curvature. By default, PyTorch's computational graph is discarded after .backward(). To compute higher-order derivatives, you need to preserve that graph so you can differentiate through the gradient computation itself.

How It Works

The magic parameter is create_graph=True in the .backward() call:

import torch

# Function: f(x) = x^3
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3

# First derivative (dy/dx)
y.backward(create_graph=True)  # Keep the graph!
first_deriv = x.grad.clone()  # dy/dx = 3x^2 = 12

# Second derivative (d²y/dx²)
x.grad.zero_()
first_deriv.backward()  # Differentiate the gradient
second_deriv = x.grad  # d²y/dx² = 6x = 12

Without create_graph=True, the first .backward() would destroy the graph, making the second derivative computation impossible.

Analogy

Think of it like filming a magician's trick. The first backward pass is watching the trick happen. With create_graph=True, you're recording it—so you can rewind and analyze how the trick was performed, enabling deeper inspection.

Key Takeaway:
Use create_graph=True in .backward() to preserve the computational graph and enable higher-order derivative computations by differentiating through the gradient calculation itself.