Limitations and the Need for Attention
What You'll Learn
You'll understand why the encoder-decoder architecture with fixed context vectors hits a wall with long sequences, and why this problem demanded a better solution.
The Fixed Context Vector Bottleneck
In the encoder-decoder architecture you've learned, the encoder compresses an entire input sequence—whether it's 5 words or 50—into a single fixed-size context vector. The decoder then tries to generate the entire output using only this one vector.
The Problem: Imagine summarizing an entire book into a single sentence, then asking someone to rewrite the book using only that sentence. That's essentially what we're asking the network to do!
Why This Fails for Long Sequences
- Information loss: A fixed-size vector (say, 512 dimensions) simply cannot capture all the nuances of a long, complex sequence
- Early token amnesia: By the time the encoder processes the 40th word, gradients from the first few words have weakened significantly
- Uniform compression: The model treats all input parts equally, with no way to focus on what's currently relevant
Real-World Impact
In machine translation, when translating "The cat sat on the mat" to French, the decoder needs different parts of the input at different times. When generating "chat" (cat), it should focus on "cat," not "mat." But with one fixed context vector, it can't selectively attend to different source words.
Enter Attention
This limitation motivated researchers to ask: What if the decoder could "look back" at the entire input sequence and decide which parts matter most at each decoding step? This question led to the attention mechanism revolution.
Key Takeaway: Fixed context vectors create an information bottleneck that worsens with sequence length, motivating attention mechanisms that let decoders dynamically focus on relevant input parts during generation.