Cross-Attention: Attending Between Different Sequences
What You'll Learn
You'll learn how cross-attention lets one sequence (like a decoder) attend to information from a completely different sequence (like an encoder), enabling powerful sequence-to-sequence modeling.
The Core Idea
Until now, we've focused on self-attention, where a sequence attends to itself—all queries, keys, and values come from the same source. Cross-attention breaks this symmetry: the queries come from one sequence, while the keys and values come from a different sequence.
Think of it like a conversation: you (the decoder) are asking questions (queries) about what someone else said (the encoder's keys and values). You're not just thinking about your own words—you're attending to theirs.
How It Works
In cross-attention:
- Queries (Q): Generated from the target sequence (e.g., decoder hidden states in translation)
- Keys (K) and Values (V): Generated from the source sequence (e.g., encoder outputs)
The attention mechanism proceeds exactly as before:
- Compute attention scores: Q · K^T
- Scale by √d_k
- Apply softmax to get attention weights
- Compute weighted sum of values
The crucial difference is the source of these vectors. This allows the decoder to "look at" relevant parts of the input sequence when generating each output token.
Example Context
In machine translation, when generating the third French word, cross-attention lets the decoder focus on relevant English words from the encoder, rather than just looking at previously generated French words.
Key Takeaway: Cross-attention enables queries from one sequence to attend to keys and values from another sequence, forming the bridge between encoder and decoder in transformer-based sequence-to-sequence models.