Course contentsShow
← Machine Learning and Deep Learning
Lesson 786 of 3,53819. PyTorch FundamentalsFree lesson

In-place Operations and Memory

Understand in-place operations (methods ending with _) and their impact on memory usage and computational graphs.

In-place Operations and Memory

What You'll Learn

You'll learn how in-place operations modify tensors directly, saving memory but affecting computational graphs in PyTorch.

The Core Idea

In PyTorch, in-place operations modify a tensor's data directly without creating a new tensor. These operations are marked by a trailing underscore (e.g., add_(), mul_(), zero_()).

Think of it like editing a document: a normal operation creates a copy, edits it, and gives you the new version (leaving the original untouched). An in-place operation opens your original document and changes it directly—no copy made.

Why Use In-place Operations?

Memory efficiency: When working with large tensors, creating copies consumes significant memory. In-place operations modify the existing data, using no additional memory for the result.

The Critical Tradeoff

PyTorch tracks operations to build computational graphs for automatic differentiation (backpropagation). In-place operations can break this tracking because they overwrite values that might be needed to compute gradients.

For example:

  • x = x + 1 creates a new tensor, preserves the old x for gradient computation
  • x.add_(1) modifies x directly, destroying the original value PyTorch might need

When to Use In-place Operations

  • Safe contexts: During inference (no gradients needed) or when you're certain the tensor isn't part of the computational graph
  • Memory-critical situations: When working with very large tensors and memory is limited
  • Explicit scenarios: Zeroing gradients with optimizer.zero_grad() or initializing weights

When to Avoid Them

  • During training: If the tensor requires gradients or is part of backpropagation
  • When unsure: The memory saved is often negligible compared to the risk of gradient errors

Key Takeaway: In-place operations (ending with _) save memory by modifying tensors directly but can disrupt gradient computation—use them cautiously during training, freely during inference.