Encoder-Decoder Architecture Overview
What you'll learn: How the two-tower Transformer design separates input processing from output generation using cross-attention.
The Two-Tower Design
The Transformer's encoder-decoder architecture splits the model into two specialized components, like having one expert who listens and understands (encoder) and another who speaks and responds (decoder).
The Encoder Tower
The encoder processes the entire input sequence in parallel. It consists of stacked layers, each containing:
- Multi-head self-attention (tokens attend to each other)
- Feed-forward networks
- Layer normalization and residual connections
The encoder builds rich, contextualized representations of the input. Since it sees the whole sequence at once, it uses bidirectional self-attention — every token can attend to every other token without restrictions.
The Decoder Tower
The decoder generates output tokens autoregressively (one at a time). Each decoder layer contains:
- Masked multi-head self-attention (causal attention for previously generated tokens)
- Cross-attention layer (the crucial bridge!)
- Feed-forward networks
- Layer normalization and residual connections
The Bridge: Cross-Attention
Cross-attention is where the magic happens. Here, the decoder:
- Uses its current state as queries
- Attends to the encoder's output as keys and values
This allows each decoder position to "look at" relevant parts of the input sequence when generating output, solving the fixed-length context vector bottleneck from earlier seq2seq models.
Information Flow
- Encoder processes full input → creates contextualized representations
- Decoder generates tokens one-by-one
- At each step, decoder cross-attends to encoder outputs
- Decoder also self-attends to its own previous outputs (masked)
Key Takeaway: The encoder-decoder Transformer uses a two-tower design where the encoder bidirectionally processes input, the decoder autoregressively generates output, and cross-attention layers connect them by letting the decoder selectively focus on relevant input positions.