Course contentsShow
← Machine Learning and Deep Learning
Lesson 1064 of 3,53824. The Transformer ArchitectureFree lesson

Cross-Attention: Attending Between Different Sequences

Understand how queries from one sequence attend to keys/values from another sequence.

Cross-Attention: Attending Between Different Sequences

What You'll Learn

You'll learn how cross-attention lets one sequence (like a decoder) attend to information from a completely different sequence (like an encoder), enabling powerful sequence-to-sequence modeling.

The Core Idea

Until now, we've focused on self-attention, where a sequence attends to itself—all queries, keys, and values come from the same source. Cross-attention breaks this symmetry: the queries come from one sequence, while the keys and values come from a different sequence.

Think of it like a conversation: you (the decoder) are asking questions (queries) about what someone else said (the encoder's keys and values). You're not just thinking about your own words—you're attending to theirs.

How It Works

In cross-attention:

  • Queries (Q): Generated from the target sequence (e.g., decoder hidden states in translation)
  • Keys (K) and Values (V): Generated from the source sequence (e.g., encoder outputs)

The attention mechanism proceeds exactly as before:

  1. Compute attention scores: Q · K^T
  2. Scale by √d_k
  3. Apply softmax to get attention weights
  4. Compute weighted sum of values

The crucial difference is the source of these vectors. This allows the decoder to "look at" relevant parts of the input sequence when generating each output token.

Example Context

In machine translation, when generating the third French word, cross-attention lets the decoder focus on relevant English words from the encoder, rather than just looking at previously generated French words.

Key Takeaway: Cross-attention enables queries from one sequence to attend to keys and values from another sequence, forming the bridge between encoder and decoder in transformer-based sequence-to-sequence models.