Bidirectional Encoders for Seq2Seq
What You'll Learn
You'll discover how bidirectional RNNs in the encoder capture richer context from input sequences by reading them both forward and backward.
The Core Idea
In standard encoder-decoder architectures, the encoder processes the input sequence from left to right (forward direction only). But language and sequences often have context that flows both ways—the meaning of a word depends on what comes before and after it.
Bidirectional encoders solve this by running two separate RNN layers over the input:
- One processes the sequence forward (left to right)
- One processes the sequence backward (right to left)
At each position, you combine the forward and backward hidden states (typically by concatenation) to create a richer representation that "knows" the full context around that position.
Why This Matters
Think of translating "bank" from English. Without context, it's ambiguous (river bank? financial bank?). A forward-only encoder at the word "bank" has only seen what came before. A bidirectional encoder sees both directions:
- Forward state: knows the left context
- Backward state: knows the right context
- Combined: understands "river bank" vs "bank account" perfectly
This creates better context vectors for the decoder, leading to more accurate sequence generation in machine translation, summarization, and other seq2seq tasks.
Implementation Pattern
The encoder now produces bidirectional hidden states at each timestep. When creating the context vector (or when using attention later), you work with these combined representations instead of single-direction states.
Key Takeaway: Bidirectional encoders process input sequences in both directions simultaneously, creating richer context-aware representations that capture the full meaning of each position before passing information to the decoder.