Course contentsShow
← Machine Learning and Deep Learning
Lesson 1052 of 3,53824. The Transformer ArchitectureFree lesson

Computing Attention Scores with Dot Products

Learn how dot products between queries and keys measure relevance between positions.

Computing Attention Scores with Dot Products

What you'll learn: How the dot product between query and key vectors mathematically measures how much two positions should "pay attention" to each other.

The Core Mechanism

You've learned that attention uses queries, keys, and values as three separate vectors. Now comes the magic: how do we actually measure relevance between positions?

The answer is surprisingly elegant: compute the dot product between a query and a key.

Why Dot Products Measure Relevance

The dot product between two vectors gives you a single number that captures their similarity:

  • High positive value: vectors point in similar directions → high relevance
  • Low or negative value: vectors are dissimilar → low relevance

Think of it like shining a flashlight (query) across a room. The dot product tells you how directly your beam hits each object (key). Objects aligned with your beam get high scores; those at odd angles get low scores.

The Mathematical Operation

For a single query vector q and a single key vector k, the attention score is:

score = q · k = q₁k₁ + q₂k₂ + ... + qₙkₙ

This is just element-wise multiplication followed by summing. In practice, when you have multiple positions, you compute all query-key pairs to create a score matrix showing how every position relates to every other position.

The dot product is fast, differentiable, and naturally captures the geometric relationship between learned representations—making it perfect for neural networks to learn meaningful attention patterns.

Key Takeaway: The dot product between query and key vectors produces a raw attention score that quantifies how relevant two positions are to each other, with higher values indicating stronger relationships that the model should focus on.