Course contentsShow
← Machine Learning and Deep Learning
Lesson 797 of 3,53819. PyTorch FundamentalsFree lesson

Non-Scalar Outputs and Gradient Arguments

Computing gradients when output is not a scalar by providing gradient argument to backward().

Non-Scalar Outputs and Gradient Arguments

What You'll Learn

You'll learn how to compute gradients when your tensor output isn't a single number, by providing a gradient argument to .backward().

The Problem with Non-Scalar Outputs

So far, you've likely computed gradients when your loss function produces a single scalar value (like a final loss number). Calling .backward() on that scalar is straightforward—PyTorch knows exactly what gradient to compute.

But what happens when your output is a vector or matrix instead of a single number? For example, if you have a tensor of shape (3,) or (2, 4), PyTorch can't automatically decide how to compute gradients—there's no single "direction" to backpropagate from.

The Solution: Gradient Arguments

To handle non-scalar outputs, you must explicitly tell .backward() which gradient to use by passing a gradient argument. This argument should be a tensor of the same shape as your output.

Think of it like this: the gradient argument represents "weights" or "importance" for each element in your output. It tells PyTorch how much each output element should contribute to the final gradients of the input parameters.

import torch

x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2  # Non-scalar output: shape (3,)

# This would error: y.backward()

# Instead, provide a gradient tensor:
gradient = torch.tensor([1.0, 1.0, 1.0])
y.backward(gradient=gradient)

print(x.grad)  # tensor([2., 2., 2.])

Here, the gradient argument [1.0, 1.0, 1.0] treats all three outputs equally, computing the gradient as if we summed them first.

Key Takeaway: When your output isn't a scalar, pass a gradient argument to .backward() with the same shape as your output to specify how each element contributes to the backpropagation.