Decoder Architecture and Conditional Generation
What you'll learn: How a decoder RNN takes the encoder's context vector and generates an output sequence one step at a time.
The Decoder's Job
You've seen how the encoder compresses an input sequence into a fixed-size context vector. Now the decoder's job is to unpack that information and generate the target sequence. Think of it like a translator who listened to a complete sentence in one language (the encoder) and now must speak it out in another language, word by word.
Step-by-Step Generation
The decoder is itself an RNN (typically an LSTM or GRU) that generates output conditionally — meaning each output depends on:
- The context vector from the encoder (usually provided as the initial hidden state)
- The previous output token (what it just generated)
- Its own hidden state (memory of what it's generated so far)
At each time step, the decoder:
- Takes the previous word it predicted (or a start token
<SOS>initially) - Combines it with its hidden state
- Produces a probability distribution over all possible next words
- Samples or selects the most likely word
- Updates its hidden state for the next step
- Continues until it produces an end token
<EOS>
Conditional Generation
This is "conditional" because the entire output sequence is conditioned on the encoder's context. Same decoder architecture, different context vector → different output sequence. For translation, the same decoder network can translate any source sentence because the context vector tells it what to translate.
Key Takeaway: The decoder RNN generates target sequences autoregressively (one token at a time), conditioned on the encoder's context vector, using each previous output to inform the next prediction.