Course contentsShow
← Machine Learning and Deep Learning
Lesson 650 of 3,53815. Neural Network FoundationsFree lesson

Detaching Tensors and Stopping Gradients

Control gradient flow by detaching tensors from the computation graph or using stop-gradient operations when needed.

Detaching Tensors and Stopping Gradients

What You'll Learn

You'll learn how to intentionally interrupt gradient flow in your computational graph to prevent certain operations from affecting backpropagation.

The Core Idea

When building neural networks, you don't always want gradients to flow through every operation. Sometimes you need to "freeze" part of your computation graph—treating certain tensors as constants rather than learnable parameters. This is where detaching comes in.

Think of your computational graph as a chain of connected nodes. Detaching a tensor is like cutting one of the links: forward computation still works normally, but when gradients flow backward during backpropagation, they stop at the cut point and don't flow any further.

Why Would You Do This?

Several practical scenarios require stopping gradients:

  • Target networks: In reinforcement learning, you compute loss against a "frozen" copy of your network
  • Semi-supervised learning: You might want gradients from labeled data only, treating predictions on unlabeled data as fixed
  • Multi-stage training: Computing auxiliary losses where you don't want gradients affecting earlier layers
  • Preventing gradient contamination: When using model outputs as pseudo-labels or reference values

How It Works

In most deep learning frameworks, tensors track their computational history. Detaching creates a new tensor that shares the same data but has no gradient tracking—it's disconnected from the graph. During backpropagation, the chain rule stops here because there's no path to follow backward.

This differs from gradient checkpointing (which trades computation for memory)—detaching is about selective gradient control.

Key Takeaway: Detaching tensors lets you control exactly where gradients flow in your network, treating specific computations as constants during backpropagation while preserving forward pass values.