Course contentsShow
← Machine Learning and Deep Learning
Lesson 646 of 3,53815. Neural Network FoundationsFree lesson

Forward Mode vs Reverse Mode Autodiff

Compare forward mode (efficient for few inputs) and reverse mode (efficient for few outputs, used in backprop).

Forward Mode vs Reverse Mode Autodiff

What you'll learn: You'll understand the two main approaches to automatic differentiation and why neural networks use reverse mode (backpropagation).

The Core Difference

Automatic differentiation can traverse a computational graph in two directions: forward mode computes derivatives as you move forward through the graph, while reverse mode computes them by sweeping backward from the output to the inputs.

Think of it like calculating trip costs: forward mode is like tracking how each gas station stop affects your total spending as you drive, while reverse mode is like working backward from your final bill to see how much each stop contributed.

Forward Mode Autodiff

Forward mode computes derivatives alongside the forward pass. For each input variable, it propagates derivative information forward through each operation. This is efficient when you have few inputs and many outputs—you make one forward sweep per input to get all its derivatives.

Efficiency: If you have 2 inputs and 1,000 outputs, forward mode needs only 2 passes.

Reverse Mode Autodiff

Reverse mode (backpropagation) computes the forward pass first, then sweeps backward from the output, calculating gradients with respect to all inputs in one pass. This is efficient when you have few outputs and many inputs.

Efficiency: If you have 1,000,000 parameters (inputs) and 1 loss value (output), reverse mode needs only 1 backward pass!

Why Neural Networks Use Reverse Mode

Neural networks typically have millions of parameters but a single scalar loss function. Reverse mode gives us gradients for all parameters in one backward sweep—exactly what we need for training.

Key Takeaway: Forward mode is efficient for few inputs/many outputs; reverse mode (backpropagation) is efficient for few outputs/many inputs, making it perfect for neural network training where we have one loss and millions of parameters.