Computing Attention Scores with Dot Products
What you'll learn: How the dot product between query and key vectors mathematically measures how much two positions should "pay attention" to each other.
The Core Mechanism
You've learned that attention uses queries, keys, and values as three separate vectors. Now comes the magic: how do we actually measure relevance between positions?
The answer is surprisingly elegant: compute the dot product between a query and a key.
Why Dot Products Measure Relevance
The dot product between two vectors gives you a single number that captures their similarity:
- High positive value: vectors point in similar directions → high relevance
- Low or negative value: vectors are dissimilar → low relevance
Think of it like shining a flashlight (query) across a room. The dot product tells you how directly your beam hits each object (key). Objects aligned with your beam get high scores; those at odd angles get low scores.
The Mathematical Operation
For a single query vector q and a single key vector k, the attention score is:
score = q · k = q₁k₁ + q₂k₂ + ... + qₙkₙ
This is just element-wise multiplication followed by summing. In practice, when you have multiple positions, you compute all query-key pairs to create a score matrix showing how every position relates to every other position.
The dot product is fast, differentiable, and naturally captures the geometric relationship between learned representations—making it perfect for neural networks to learn meaningful attention patterns.
Key Takeaway: The dot product between query and key vectors produces a raw attention score that quantifies how relevant two positions are to each other, with higher values indicating stronger relationships that the model should focus on.