GRU Reset and Update Gates
What you'll learn: How GRU gates control which information flows from the past and how much new information to accept.
The Two-Gate Design
While LSTMs use three gates and a separate cell state, GRUs simplify things with just two gates: the reset gate and the update gate. These two mechanisms work together to manage memory without the complexity of LSTM's triple-gate system.
The Reset Gate: Forgetting the Past
The reset gate determines how much of the previous hidden state should be "forgotten" when computing the new candidate hidden state. Think of it like deciding whether to start fresh or build on your recent memory.
- When the reset gate outputs values close to 0, it effectively ignores most of the previous hidden state
- When it outputs values close to 1, it fully incorporates the past information
- This lets the GRU drop irrelevant context when encountering something new
Analogy: Like selectively forgetting irrelevant details from a previous conversation when a new topic begins.
The Update Gate: Balancing Old vs New
The update gate decides the balance between keeping the old hidden state versus accepting new candidate information. It's a single gate that performs what would require both LSTM's forget and input gates.
- Values close to 0 → mostly accept new information (forget the old)
- Values close to 1 → mostly keep old information (ignore the new)
- It creates a weighted blend:
h_new = (1 - z) * h_old + z * h_candidate
Analogy: Like a DJ's crossfader, smoothly blending between the old track (previous state) and the new track (candidate state).
How They Work Together
The reset gate filters the past before computing what's new, while the update gate blends old and new after computing the candidate. This two-step process provides flexible memory control with fewer parameters than LSTM.
Key Takeaway: The reset gate controls access to previous hidden state when forming new memories, while the update gate balances how much old versus new information to retain in the final hidden state.