Encoder-Decoder Architecture Fundamentals
What you'll learn: How the encoder-decoder structure enables RNNs to transform variable-length input sequences into variable-length output sequences.
The Two-Component Structure
The encoder-decoder architecture is a fundamental design pattern that solves a key challenge: how do we map an input sequence of one length to an output sequence of a potentially different length? This is the backbone of tasks like machine translation (English sentence → French sentence) or text summarization (long article → short summary).
How It Works
The architecture consists of two main components:
-
Encoder: An RNN (typically LSTM or GRU) that processes the entire input sequence step-by-step and compresses all the information into a fixed-size context vector (also called a "thought vector"). This context vector is usually the final hidden state of the encoder RNN.
-
Decoder: Another RNN that takes the context vector and generates the output sequence one element at a time. The decoder is initialized with the encoder's context vector and produces outputs sequentially until a stop signal is reached.
The Translation Analogy
Think of it like a translator who first reads an entire English paragraph (encoding it into understanding), then writes out the French translation word-by-word based on that understanding (decoding). The translator doesn't translate word-by-word simultaneously—they first comprehend the full meaning, then produce the translation.
Key Characteristics
- Variable-length handling: Input can be 5 words, output can be 8 words
- Information bottleneck: All input information must flow through the context vector
- Sequential generation: Decoder produces outputs one step at a time
- Many-to-many architecture: Combines the encoder (many-to-one) with decoder (one-to-many)
Key Takeaway: The encoder-decoder architecture uses two RNNs—one to compress a variable-length input into a fixed context vector, and another to expand that context into a variable-length output sequence.