Forward Propagation in RNNs
You'll learn how an RNN processes a sequence step-by-step, computing hidden states and outputs from start to finish.
The Core Idea
Forward propagation in RNNs is like reading a sentence word by word while maintaining a running summary of what you've read so far. At each time step, the network takes the current input and the previous hidden state, combines them, and produces both a new hidden state (your updated summary) and an output.
Step-by-Step Computation
When you unroll an RNN through time, forward propagation proceeds sequentially:
- Initialize: Start with an initial hidden state h₀ (often zeros)
- For each time step t:
- Take current input xₜ
- Combine it with previous hidden state hₜ₋₁
- Apply the recurrence relation: hₜ = tanh(Wₕ·hₜ₋₁ + Wₓ·xₜ + b)
- Compute output: yₜ = Wᵧ·hₜ + bᵧ
- Repeat until sequence ends
The Flow Analogy
Think of it like a relay race where each runner (time step) receives a baton (hidden state) from the previous runner, combines it with their own contribution (current input), and passes an updated baton to the next runner. The final positions at each checkpoint are your outputs.
The hidden state acts as the network's memory, carrying information forward through the sequence. This allows the RNN to maintain context—for example, understanding that "bank" means something different in "river bank" versus "bank account" based on earlier words.
Key Takeaway: Forward propagation in RNNs processes sequences left-to-right, computing hidden states through the recurrence relation hₜ = f(hₜ₋₁, xₜ), where each step builds on the previous one's hidden state to maintain sequential context.