Course contentsShow
← Machine Learning and Deep Learning
Lesson 1093 of 3,53824. The Transformer ArchitectureFree lesson

Encoder-Decoder Architecture Overview

Understanding the two-tower design: encoder processes input, decoder generates output with cross-attention.

Encoder-Decoder Architecture Overview

What you'll learn: How the two-tower Transformer design separates input processing from output generation using cross-attention.

The Two-Tower Design

The Transformer's encoder-decoder architecture splits the model into two specialized components, like having one expert who listens and understands (encoder) and another who speaks and responds (decoder).

The Encoder Tower

The encoder processes the entire input sequence in parallel. It consists of stacked layers, each containing:

  • Multi-head self-attention (tokens attend to each other)
  • Feed-forward networks
  • Layer normalization and residual connections

The encoder builds rich, contextualized representations of the input. Since it sees the whole sequence at once, it uses bidirectional self-attention — every token can attend to every other token without restrictions.

The Decoder Tower

The decoder generates output tokens autoregressively (one at a time). Each decoder layer contains:

  • Masked multi-head self-attention (causal attention for previously generated tokens)
  • Cross-attention layer (the crucial bridge!)
  • Feed-forward networks
  • Layer normalization and residual connections

The Bridge: Cross-Attention

Cross-attention is where the magic happens. Here, the decoder:

  • Uses its current state as queries
  • Attends to the encoder's output as keys and values

This allows each decoder position to "look at" relevant parts of the input sequence when generating output, solving the fixed-length context vector bottleneck from earlier seq2seq models.

Information Flow

  1. Encoder processes full input → creates contextualized representations
  2. Decoder generates tokens one-by-one
  3. At each step, decoder cross-attends to encoder outputs
  4. Decoder also self-attends to its own previous outputs (masked)

Key Takeaway: The encoder-decoder Transformer uses a two-tower design where the encoder bidirectionally processes input, the decoder autoregressively generates output, and cross-attention layers connect them by letting the decoder selectively focus on relevant input positions.