LSTM Input Gate and Candidate Values
What you'll learn: How LSTMs decide what new information to add to memory and combine it with freshly generated candidate values.
The Input Gate's Role
After the forget gate decides what to discard from memory, the LSTM needs to determine what new information to add. This is where the input gate comes in. Think of it like a bouncer at a club—not every piece of new information deserves to enter your long-term memory. The input gate produces values between 0 and 1 that act as filters for new content.
Candidate Values: The Raw Update
Separately, the LSTM creates candidate values using a tanh activation. These candidates represent the potential new information that could be added to the cell state. The tanh produces values between -1 and 1, allowing the network to propose both positive and negative updates to memory.
Think of candidate values as rough draft notes—they're what could be remembered, but haven't been vetted yet.
Combining Gate and Candidates
The magic happens when you multiply the input gate's output with the candidate values. The input gate acts as a selective filter:
- If the input gate outputs ~1 for a dimension, that candidate value passes through fully
- If it outputs ~0, that candidate is blocked
- Values in between allow partial updates
This filtered result is then added to the cell state (after the forget gate has done its job), updating the LSTM's memory with relevant new information.
Analogy
Imagine taking notes during a lecture. The candidate values are everything the professor says. The input gate is your judgment about what's important enough to write down. You only add to your notebook what passes both criteria: said AND deemed important.
Key Takeaway: The input gate selectively filters candidate values (generated by tanh) to determine exactly what new information gets added to the LSTM's cell state memory.