Backpropagation Through Time (BPTT)
What you'll learn: How gradients flow backward through the unrolled RNN network and accumulate across all time steps to enable training.
The Challenge of Training RNNs
Remember when we unrolled an RNN through time? Each time step became like a layer in a deep network, all sharing the same weights. Now we need to train those weights—but how do we compute gradients when the same parameters are used repeatedly across multiple time steps?
How BPTT Works
Backpropagation Through Time treats the unrolled RNN as a special deep network and applies the chain rule backward through all time steps.
Think of it like tracing responsibility backward through a chain of events. If your final prediction at time step t=3 is wrong, we ask: "How did each previous time step (t=2, t=1, t=0) contribute to this error?" The gradient flows backward from the output through each time step, accumulating contributions along the way.
The Accumulation Process
Here's the key insight: because the same weight matrices are used at every time step, gradients from all time steps add up. When we compute the gradient for a weight matrix, we sum contributions from every point in time where that weight influenced the output.
Analogy: Imagine a relay race where the same runner appears multiple times. If the team loses, you'd evaluate that runner's performance in each leg of the race, then sum up all their mistakes to understand their total impact.
The Flow Pattern
Gradients flow backward from time t to t-1, then t-2, and so on, all the way back to the beginning of the sequence. At each step, we compute how the hidden state and parameters affected the loss, accumulating these gradients as we go.
Key Takeaway: BPTT computes gradients by flowing backward through the unrolled RNN, accumulating contributions from all time steps where shared weights influenced the final prediction—enabling us to update those weights through standard gradient descent.