LSTM Output Gate and Hidden State
What you'll learn: How the LSTM output gate decides what information from the cell state becomes the final hidden state output at each timestep.
The Final Filter
You've seen how the forget gate clears old memories and the input gate writes new ones into the cell state. But there's one crucial step left: what do we actually output? The cell state contains everything the LSTM remembers, but we don't always want to expose all of it. The output gate acts as a smart filter, deciding which parts of the cell's memory are relevant right now.
How the Output Gate Works
The output gate uses a sigmoid activation to create another filter (values between 0 and 1):
o_t = sigmoid(W_o · [h_{t-1}, x_t] + b_o)
But here's the key: the output gate doesn't filter the raw cell state. First, the cell state passes through a tanh activation (squashing values to between -1 and 1), then the output gate filters it:
h_t = o_t ⊙ tanh(c_t)
Why Two Steps?
Think of it like a library. The cell state c_t is your entire collection of books (memories). The tanh squashes these into a readable format, and the output gate o_t decides which books you actually pull off the shelf to show someone. You might have knowledge about many topics, but you only share what's relevant to the current conversation.
The hidden state h_t is what gets passed to the next timestep and what's often used as the LSTM's output for predictions.
Key Takeaway: The output gate filters a tanh-transformed cell state to produce the hidden state, controlling what information the LSTM exposes at each timestep while keeping the full memory intact internally.