Course contentsShow
← Machine Learning and Deep Learning
Lesson 1053 of 3,53824. The Transformer ArchitectureFree lesson

The Attention Score Matrix

Visualize and compute the matrix of all pairwise attention scores between sequence positions.

The Attention Score Matrix

What you'll learn: How to organize all pairwise attention scores between sequence positions into a single matrix structure.

Understanding the Matrix Layout

In the previous lesson, you learned to compute attention scores using dot products between queries and keys. When you have a sequence of multiple positions (like words in a sentence), you need scores for every query position attending to every key position. This creates a matrix of scores.

Think of it like a spreadsheet where:

  • Each row represents one query position (what's currently "asking")
  • Each column represents one key position (what's being "looked at")
  • Each cell contains the attention score between that query-key pair

The Computation

For a sequence of length n, you have:

  • n query vectors (one per position)
  • n key vectors (one per position)

Instead of computing scores one-by-one, you stack all queries into a matrix Q and all keys into a matrix K. The attention score matrix is simply:

Score Matrix = Q · K^T

(Matrix multiplication of Q with the transpose of K)

This gives you an n × n matrix where entry (i, j) tells you how much position i should attend to position j.

Example

For a 3-word sequence, you get a 3×3 matrix. If word 1 strongly relates to word 3, the score at position (1, 3) will be high.

Key Takeaway: The attention score matrix efficiently computes all pairwise attention scores in one matrix operation, creating a complete "attention map" showing how every position relates to every other position in the sequence.