The requires_grad Flag
What you'll learn: How to tell PyTorch which tensors need gradient tracking so you can train your models efficiently.
The Core Idea
In PyTorch, not every tensor needs to have its gradients computed. The requires_grad flag is a boolean attribute that marks whether a tensor should track operations performed on it for automatic differentiation. When requires_grad=True, PyTorch builds a computational graph behind the scenes, recording every operation so it can later compute gradients via backpropagation.
Think of requires_grad as a "breadcrumb trail" switch. When turned on, PyTorch drops breadcrumbs at each operation, allowing it to retrace steps backward and calculate how much each input contributed to the final output.
When to Enable It
You typically enable requires_grad for:
- Model parameters (weights and biases) – these are what you're training
- Inputs you want to analyze – for visualization or adversarial examples
You keep it disabled (the default) for:
- Training data inputs – you don't update your dataset
- Constants and hyperparameters – these aren't learned
- Inference/evaluation – saves memory and speeds up computation
How It Works
When you create a tensor, it defaults to requires_grad=False. To enable tracking:
x = torch.tensor([2.0, 3.0], requires_grad=True)
Or enable it after creation:
x = torch.tensor([2.0, 3.0])
x.requires_grad = True
Any tensor resulting from operations on a requires_grad=True tensor will also track gradients automatically.
Key Takeaway: The requires_grad flag tells PyTorch which tensors need gradient computation—enable it for parameters you want to train, disable it everywhere else to save memory and computation.