Course contentsShow
← Machine Learning and Deep Learning
Lesson 795 of 3,53819. PyTorch FundamentalsFree lesson

Detaching Tensors from the Graph

Using .detach() and torch.no_grad() to prevent gradient tracking when computing intermediate values.

Detaching Tensors from the Graph

What you'll learn: How to selectively prevent gradient tracking when you need intermediate values without affecting backpropagation.


Sometimes during training, you need to compute values that shouldn't participate in gradient calculations—maybe for logging, visualization, or conditional logic. If you don't stop gradient tracking, PyTorch will waste memory building unnecessary graph nodes and may compute incorrect gradients.

The Core Problem

When a tensor has requires_grad=True, every operation on it builds part of the computational graph. But what if you want to use that tensor's value without extending the graph? That's where detaching comes in.

Two Ways to Stop Tracking

Using .detach()
Creates a new tensor that shares the same data but is disconnected from the graph. Changes to the original won't affect the detached version's gradient history.

x = torch.tensor([2.0], requires_grad=True)
y = x ** 2

# Detach y from the graph
y_detached = y.detach()
# y_detached has the value 4.0, but no gradient tracking

Using torch.no_grad() context
Temporarily disables gradient tracking for all operations inside the block. More efficient when you have multiple operations.

x = torch.tensor([2.0], requires_grad=True)

with torch.no_grad():
    y = x ** 2  # This operation doesn't build graph nodes
    z = y * 3   # Neither does this

When to Use Each

Use .detach() for single tensors you want to "snapshot." Use torch.no_grad() when computing multiple operations that don't need gradients (like during validation or generating predictions).

Key Takeaway: Detaching tensors with .detach() or torch.no_grad() prevents unnecessary gradient tracking, saving memory and ensuring only intended operations participate in backpropagation.