Detaching Tensors from the Graph
What you'll learn: How to selectively prevent gradient tracking when you need intermediate values without affecting backpropagation.
Sometimes during training, you need to compute values that shouldn't participate in gradient calculations—maybe for logging, visualization, or conditional logic. If you don't stop gradient tracking, PyTorch will waste memory building unnecessary graph nodes and may compute incorrect gradients.
The Core Problem
When a tensor has requires_grad=True, every operation on it builds part of the computational graph. But what if you want to use that tensor's value without extending the graph? That's where detaching comes in.
Two Ways to Stop Tracking
Using .detach()
Creates a new tensor that shares the same data but is disconnected from the graph. Changes to the original won't affect the detached version's gradient history.
x = torch.tensor([2.0], requires_grad=True)
y = x ** 2
# Detach y from the graph
y_detached = y.detach()
# y_detached has the value 4.0, but no gradient tracking
Using torch.no_grad() context
Temporarily disables gradient tracking for all operations inside the block. More efficient when you have multiple operations.
x = torch.tensor([2.0], requires_grad=True)
with torch.no_grad():
y = x ** 2 # This operation doesn't build graph nodes
z = y * 3 # Neither does this
When to Use Each
Use .detach() for single tensors you want to "snapshot." Use torch.no_grad() when computing multiple operations that don't need gradients (like during validation or generating predictions).
Key Takeaway: Detaching tensors with .detach() or torch.no_grad() prevents unnecessary gradient tracking, saving memory and ensuring only intended operations participate in backpropagation.