LSTM Forget Gate
What You'll Learn
You'll learn how the forget gate selectively erases outdated information from the LSTM cell state using a sigmoid-gated multiplication operation.
The Problem of Irrelevant Memory
Imagine reading a story where characters change from chapter to chapter. Your brain naturally "forgets" details about old characters when new ones appear—you don't need to remember everything forever. LSTMs need the same ability: a mechanism to actively decide what to forget from the cell state.
How the Forget Gate Works
The forget gate is the first gate that processes information in each LSTM time step. Here's the step-by-step process:
-
Input combination: The gate receives two inputs—the current input
x_tand the previous hidden stateh_{t-1} -
Sigmoid activation: These inputs pass through a sigmoid function (σ), producing values between 0 and 1:
f_t = σ(W_f · [h_{t-1}, x_t] + b_f) -
Element-wise multiplication: The forget gate output
f_tmultiplies the previous cell stateC_{t-1}element-by-element -
Selective forgetting:
- Values close to 1 mean "keep this information"
- Values close to 0 mean "forget this information"
- Values in between partially retain information
An Analogy
Think of the cell state as a whiteboard with notes. The forget gate is like an eraser with variable transparency—it can completely erase parts (0), leave other parts untouched (1), or partially fade sections (values between 0 and 1).
Key Takeaway: The forget gate uses sigmoid activation to produce values between 0 and 1, which multiply the cell state element-wise, allowing the LSTM to selectively remove outdated information while preserving relevant memories.