Course contentsShow
← Machine Learning and Deep Learning
Lesson 1013 of 3,53823. Recurrent Neural NetworksFree lesson

LSTM Architecture Overview

The structure of Long Short-Term Memory cells: cell state, hidden state, and the three gates that control information flow.

LSTM Architecture Overview

What you'll learn: You'll understand the internal components of an LSTM cell and how gates control information flow to solve the vanishing gradient problem.

The Need for LSTM

Regular RNNs struggle with long-term dependencies because gradients vanish during backpropagation. Long Short-Term Memory (LSTM) cells solve this by introducing a sophisticated gating mechanism that carefully controls what information to remember, forget, and output.

The Four Core Components

1. Cell State (the Memory Highway)

Think of the cell state as a conveyor belt running straight through the network. Information can flow along it relatively unchanged, allowing gradients to flow backward without vanishing. This is the "long-term memory" of the LSTM.

2. Hidden State (the Output)

The hidden state is what gets passed to the next time step and serves as the LSTM's output. It's a filtered version of the cell state, containing only the information relevant right now.

3. The Three Gates

Forget Gate: Decides what information to throw away from the cell state. It looks at the previous hidden state and current input, then outputs a number between 0 (forget completely) and 1 (keep entirely) for each value in the cell state.

Input Gate: Decides what new information to store in the cell state. It has two parts: one decides which values to update, another creates candidate values that could be added.

Output Gate: Decides what to output based on the cell state. It filters the cell state to produce the hidden state that goes to the next time step.

These gates use sigmoid activations (outputting 0-1) to control information flow, acting like valves that can be opened or closed.

Key Takeaway: LSTMs use a cell state highway and three gates (forget, input, output) to carefully regulate information flow, enabling networks to learn long-term dependencies without gradient vanishing.