Context Vector as Bottleneck
What you'll learn: You'll understand why the fixed-size context vector in encoder-decoder architectures creates an information compression problem that limits model performance.
The Core Problem
In the encoder-decoder architecture you just learned, the encoder processes an entire input sequence (like a sentence) and compresses everything into a single fixed-dimensional vector—the context vector. This vector must capture all the meaningful information from the input, regardless of whether the input is 5 words or 50 words long.
This is the bottleneck problem: forcing all information through a single fixed-size vector, no matter how long or complex the input sequence is.
An Analogy
Imagine trying to summarize an entire novel using exactly 10 sentences—no more, no less. Whether it's a short story or War and Peace, you're stuck with those same 10 sentences. Important details get lost, nuances disappear, and the summary becomes increasingly inadequate as the source material grows longer.
Why This Matters
The context vector has fixed dimensions (say, 512 or 1024 values). When encoding:
- Short sequences: The context vector may be sufficient
- Long sequences: Critical information gets squeezed out or overwritten as later inputs update the encoder's hidden state
- Complex relationships: Subtle dependencies between distant words become nearly impossible to preserve
This bottleneck is especially problematic for tasks like machine translation, where the decoder must reconstruct detailed output from this compressed representation alone.
Key Takeaway: The context vector bottleneck forces all input information through a fixed-size representation, causing information loss that worsens with longer sequences—a fundamental limitation that motivated the development of attention mechanisms.