Accessing Gradients with .grad
What you'll learn: How to read the computed gradients from PyTorch tensors after calling .backward() and what happens when gradients accumulate.
The Core Idea
After you've built a computational graph and called .backward() to compute gradients, you need to actually access those gradients to update your model parameters. PyTorch stores the computed gradient of each tensor in its .grad attribute.
Think of .grad as a "results mailbox" attached to each tensor. When you call .backward(), PyTorch performs backpropagation through the computational graph and delivers the computed gradient to each tensor's .grad attribute. You can then read from this mailbox to see how much each parameter contributed to the loss.
Reading Gradients
import torch
x = torch.tensor([2.0], requires_grad=True)
y = x ** 2
y.backward()
# Access the gradient
print(x.grad) # tensor([4.0])
The gradient tells us: "If you increase x by a tiny amount, y will increase by approximately 4 times that amount."
Gradient Accumulation
Important: Gradients accumulate by default. If you call .backward() multiple times without clearing gradients, they add up:
x = torch.tensor([2.0], requires_grad=True)
y = x ** 2
y.backward()
print(x.grad) # tensor([4.0])
z = x ** 3
z.backward()
print(x.grad) # tensor([16.0]) — accumulated! (4.0 + 12.0)
This accumulation is intentional and useful for batch processing, but you typically need to zero gradients between optimization steps.
Key Takeaway: After calling .backward(), read computed gradients from the .grad attribute of your tensors; remember that gradients accumulate across multiple backward passes unless explicitly zeroed.