Cross-Attention vs. Self-Attention Heads
What You'll Learn
You'll understand how multi-head attention operates differently when connecting encoder and decoder sequences versus attending within a single sequence.
The Core Distinction
In encoder-decoder architectures (like the original Transformer for translation), self-attention heads and cross-attention heads serve fundamentally different purposes, even though they use the same underlying attention mechanism.
Self-Attention Heads
These operate within a single sequence:
- Encoder self-attention: Each word in the source sentence attends to all other source words
- Decoder self-attention: Each word in the target sentence attends to previous target words (with causal masking)
- Purpose: Capture relationships and context within one sequence
Cross-Attention Heads
These bridge two different sequences:
- Queries come from the decoder (target sequence)
- Keys and Values come from the encoder (source sequence)
- Purpose: Align target words with relevant source words for translation/generation
The Analogy
Think of translating a book:
- Self-attention is like understanding grammar and word relationships within the English text itself
- Cross-attention is like looking back at the English text while writing the French translation to find which English words correspond to the French word you're currently generating
How They Differ in Practice
In a decoder layer, you typically see:
- Masked self-attention on decoder inputs (target attends to target)
- Cross-attention using encoder outputs (target attends to source)
- Feed-forward network
The cross-attention layer's keys and values come from the encoder's final output, while queries come from the decoder's own processing—enabling source-target alignment.
Key Takeaway: Self-attention heads learn relationships within one sequence, while cross-attention heads learn alignment between two sequences—crucial for encoder-decoder tasks like translation where source and target differ.