Course contentsShow
← Machine Learning and Deep Learning
Lesson 1098 of 3,53824. The Transformer ArchitectureFree lesson

Information Flow Through Encoder-Decoder

Tracing how input representations flow through encoder, then guide decoder via cross-attention.

Information Flow Through Encoder-Decoder

What You'll Learn

You'll learn how data travels from input through the encoder, becomes a rich representation, and then guides the decoder to generate output via cross-attention.

The Complete Journey

Now that you understand all the components, let's trace the complete path information takes through a Transformer encoder-decoder model.

Step 1: Encoding the Input

Your input sequence (e.g., "Hello world") is converted to embeddings, combined with positional encodings, and fed into the encoder stack. Each encoder layer applies:

  • Self-attention (tokens attend to each other)
  • Feed-forward processing

The output is a contextualized representation of every input token—each token's vector now contains information about the entire input sequence.

Step 2: Decoder Self-Attention

The decoder processes the target sequence (e.g., "Bonjour monde" during training). First, masked self-attention allows each position to attend only to previous positions, maintaining the autoregressive property.

Step 3: Cross-Attention Connection

Here's where encoder and decoder connect! The decoder's cross-attention layer uses:

  • Query: from the decoder's current representation
  • Keys and Values: from the encoder's final output

This lets each decoder position "look at" the entire encoded input sequence, deciding which input tokens are relevant for generating the current output token.

Step 4: Generation

The decoder continues through its feed-forward layer and outputs predictions, with cross-attention providing the crucial link back to the source information at every decoder layer.

Analogy: Think of the encoder as a librarian who reads and summarizes all books (input). The decoder is a writer who, while writing each sentence, constantly asks the librarian "what did the books say about X?" via cross-attention.

Key Takeaway: Information flows from input → encoder (self-attention creates context) → decoder (cross-attention retrieves relevant input context) → output, with cross-attention serving as the bridge between source and target sequences.