LSTM Forward Pass Equations
What You'll Learn
You'll learn the complete mathematical formulation that ties together all four LSTM computations into a single forward pass through time.
The Complete LSTM Picture
You've learned each gate individually—now let's see how they work together in one unified forward pass. An LSTM cell takes an input x_t and previous hidden state h_{t-1}, then produces a new hidden state h_t and cell state c_t through four coordinated steps.
The Four Equations in Order
Think of these equations as a recipe executed in sequence at each time step:
1. Forget Gate
f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
Decides what to keep from the old cell state.
2. Input Gate and Candidate
i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
c̃_t = tanh(W_c · [h_{t-1}, x_t] + b_c)
Determines what new information to add.
3. Cell State Update
c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t
Combines old memories (filtered by forget) with new candidates (filtered by input).
4. Output Gate and Hidden State
o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
h_t = o_t ⊙ tanh(c_t)
Produces the final output for this time step.
The Flow
These equations execute sequentially: each gate reads the same inputs [h_{t-1}, x_t] but uses different weight matrices. The ⊙ symbol means element-wise multiplication. The σ is the sigmoid function (outputs 0-1 for gating), while tanh outputs -1 to 1 for actual values.
Key Takeaway: The LSTM forward pass combines four parameterized transformations—forget gate, input gate with candidate, cell update, and output gate—to create a memory cell that can learn long-term dependencies while avoiding vanishing gradients.