The Basic RNN Cell
What you'll learn: How a single RNN cell processes sequential data by maintaining and updating a hidden state over time.
The Core Idea
Unlike feedforward networks that process each input independently, a Recurrent Neural Network (RNN) cell has memory. It takes two inputs at each time step: the current data point and a hidden state from the previous step. This hidden state acts like the network's "memory" of what it has seen so far.
Think of reading a sentence word-by-word. When you read "The cat sat on the...", your brain doesn't process "the" in isolation—it remembers the previous words to understand context. An RNN cell works similarly: it combines new information with its memory of the past.
The Recurrent Computation
At each time step t, the RNN cell performs this update:
h_t = f(W_h · h_{t-1} + W_x · x_t + b)
Where:
h_tis the new hidden state (the updated memory)h_{t-1}is the previous hidden statex_tis the current inputW_handW_xare weight matricesfis an activation function (typically tanh)
Weight Sharing Across Time
Crucially, the same weights (W_h and W_x) are used at every time step. This is called weight sharing—the RNN doesn't learn different rules for position 1 vs position 10. It learns one set of rules that apply throughout the sequence.
The Feedback Loop
The magic happens because h_t becomes the input for the next step. This creates a feedback loop: output feeds back as input, allowing information to persist and influence future predictions.
Key Takeaway: An RNN cell maintains a hidden state that gets updated at each time step using the same shared weights, creating a feedback loop that allows the network to remember and use information from earlier in the sequence.