Backward Pass and Gradient Accumulation
What You'll Learn
You'll learn how to systematically compute gradients by traversing a computational graph backwards, accumulating contributions from all paths.
The Core Idea
Once you've built a computational graph and completed the forward pass, the backward pass travels through that graph in reverse topological order—from output back to inputs. At each node, you compute how the output changes with respect to that node's inputs (local gradients), then multiply by the gradient flowing backward from downstream nodes (the chain rule). These gradients are accumulated when multiple paths converge at a node.
Why Reverse Order?
Think of water flowing downhill through a network of channels. To understand how much water originated from each source, you'd trace backwards from the ocean, tracking contributions through each junction. Similarly, to find how each parameter affects the final loss, we trace gradients backward from the loss node.
Gradient Accumulation in Action
When a variable feeds into multiple operations (e.g., a weight matrix used in several layers or a node that branches), gradients from all downstream paths must be summed. This is gradient accumulation: each backward path contributes its gradient, and the total derivative respects all the ways that variable influences the final output.
For example, if parameter w affects the loss through two different branches, both branches contribute gradients, and we add them: dL/dw = dL/dw_path1 + dL/dw_path2.
The Algorithm Flow
- Complete forward pass, storing intermediate values
- Initialize gradient at loss node (usually
dL/dL = 1) - Visit nodes in reverse topological order
- At each node: compute local gradients, multiply by incoming gradient, accumulate if multiple paths exist
- Store accumulated gradients for parameter updates
Key Takeaway: The backward pass traverses the computational graph in reverse, computing local gradients and accumulating contributions from all paths to correctly calculate how each parameter affects the final loss.