Course contentsShow
← Machine Learning and Deep Learning
Lesson 1078 of 3,53824. The Transformer ArchitectureFree lesson

Cross-Attention vs. Self-Attention Heads

How multi-head attention is used differently in encoder-decoder architectures for source-target alignment.

Cross-Attention vs. Self-Attention Heads

What You'll Learn

You'll understand how multi-head attention operates differently when connecting encoder and decoder sequences versus attending within a single sequence.

The Core Distinction

In encoder-decoder architectures (like the original Transformer for translation), self-attention heads and cross-attention heads serve fundamentally different purposes, even though they use the same underlying attention mechanism.

Self-Attention Heads

These operate within a single sequence:

  • Encoder self-attention: Each word in the source sentence attends to all other source words
  • Decoder self-attention: Each word in the target sentence attends to previous target words (with causal masking)
  • Purpose: Capture relationships and context within one sequence

Cross-Attention Heads

These bridge two different sequences:

  • Queries come from the decoder (target sequence)
  • Keys and Values come from the encoder (source sequence)
  • Purpose: Align target words with relevant source words for translation/generation

The Analogy

Think of translating a book:

  • Self-attention is like understanding grammar and word relationships within the English text itself
  • Cross-attention is like looking back at the English text while writing the French translation to find which English words correspond to the French word you're currently generating

How They Differ in Practice

In a decoder layer, you typically see:

  1. Masked self-attention on decoder inputs (target attends to target)
  2. Cross-attention using encoder outputs (target attends to source)
  3. Feed-forward network

The cross-attention layer's keys and values come from the encoder's final output, while queries come from the decoder's own processing—enabling source-target alignment.

Key Takeaway: Self-attention heads learn relationships within one sequence, while cross-attention heads learn alignment between two sequences—crucial for encoder-decoder tasks like translation where source and target differ.