Course contentsShow
← Machine Learning and Deep Learning
Lesson 1096 of 3,53824. The Transformer ArchitectureFree lesson

Cross-Attention Mechanism

How decoder attends to encoder outputs: queries from decoder, keys and values from encoder.

Cross-Attention Mechanism

What You'll Learn

You'll learn how the decoder attends to encoder outputs using queries from the decoder and keys/values from the encoder.

The Core Idea

In an encoder-decoder Transformer, cross-attention is where the magic of connecting input and output sequences happens. Unlike self-attention (where a sequence attends to itself), cross-attention allows the decoder to "look at" the encoder's outputs while generating each token.

Here's how it works:

The Three Vectors Come From Different Places

In cross-attention:

  • Queries (Q): Come from the decoder's current hidden states (what we're trying to generate)
  • Keys (K): Come from the encoder's outputs (the input we're translating/processing from)
  • Values (V): Also come from the encoder's outputs

An Analogy

Think of translating English to French:

  • The decoder (currently writing French) asks queries like "What part of the English sentence is relevant now?"
  • The encoder provides keys (pointers to English words) and values (their meanings)
  • The decoder matches its query against all keys to find which English words to focus on
  • It then takes a weighted sum of those values to inform the current French word

The Attention Flow

  1. Decoder generates queries from its current state
  2. These queries compute attention scores with encoder keys
  3. Softmax normalizes these scores into weights
  4. Weighted sum of encoder values creates the context
  5. This context helps the decoder generate the next token

Key Takeaway: Cross-attention bridges encoder and decoder by letting decoder queries attend to encoder keys and values, enabling the model to selectively focus on relevant input information when generating each output token.