Backpropagation Through Time: RNN Preview
What you'll learn: How backpropagation adapts to handle recurrent neural networks that process sequences by "unrolling" the network across time steps.
The Challenge of Sequential Data
So far, we've seen backpropagation work through layers stacked in space—one layer feeding into the next. But what happens when your network processes sequences, like words in a sentence or prices over time? Recurrent Neural Networks (RNNs) reuse the same layer at multiple time steps, feeding outputs back as inputs for the next step.
Unrolling Through Time
Backpropagation Through Time (BPTT) handles this by conceptually "unrolling" the recurrent network into a deep feedforward network where each time step becomes its own layer. Imagine a sentence with 5 words: we create 5 copies of the same RNN layer, one for each word, connected sequentially.
How It Works
During the forward pass, the network processes input step-by-step through time. During backpropagation, gradients flow backward through these unrolled time steps—from the final output all the way back to the first input. The chain rule still applies, but now the chain extends horizontally through time as well as vertically through layers.
Think of it like a video recording: forward propagation plays the video forward, frame by frame. BPTT rewinds the video, calculating how each frame's changes affect the final result.
The Key Difference
Unlike standard backpropagation through independent layers, BPTT must accumulate gradients across all time steps where the same weights are used. The gradient for a weight includes contributions from every time step it participated in.
Key Takeaway: Backpropagation Through Time extends the chain rule to sequential data by unrolling recurrent connections across time steps, enabling gradient computation through sequences while reusing the same weights at each step.