GRU Forward Pass Equations
What You'll Learn
You'll learn the exact mathematical formulas that govern how a GRU cell computes its output at each time step.
The Core Idea
While you've seen the conceptual role of the reset and update gates in GRUs, now we'll formalize exactly how these gates combine to produce the hidden state. The GRU forward pass involves three sequential computations that work together to selectively update memory.
The Three Equations
1. Reset Gate:
r_t = σ(W_r · [h_{t-1}, x_t] + b_r)
The reset gate decides how much of the previous hidden state h_{t-1} to "forget" when computing the candidate state. The sigmoid σ outputs values between 0 (forget completely) and 1 (remember fully).
2. Candidate Hidden State:
h̃_t = tanh(W_h · [r_t ⊙ h_{t-1}, x_t] + b_h)
This is the "proposed" new memory. Notice how r_t ⊙ h_{t-1} (element-wise multiplication) applies the reset gate—when r_t is near 0, the previous state is effectively ignored, allowing the GRU to start fresh.
3. Update Gate and Final Hidden State:
z_t = σ(W_z · [h_{t-1}, x_t] + b_z)
h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t
The update gate z_t interpolates between keeping the old state h_{t-1} and accepting the candidate h̃_t. Think of it as a smoothly adjustable slider: when z_t = 0, we keep the old state entirely; when z_t = 1, we fully adopt the new candidate.
The Flow
Input and previous state → reset gate decides what to forget → compute candidate → update gate blends old and new → output new hidden state.
Key Takeaway: The GRU forward pass uses three equations—reset gate for selective memory access, candidate state for proposed updates, and update gate for interpolating between old and new states—making it simpler than LSTM while still handling long-term dependencies.