Query, Key, Value: The Three Vectors
What you'll learn: How attention mechanisms use three distinct vectors—query, key, and value—to compute weighted representations of input sequences.
The Core Idea
In attention mechanisms, we transform our input data into three different vectors with distinct roles:
- Query (Q): What you're looking for
- Key (K): What each item offers as an identifier
- Value (V): The actual content to retrieve
Think of it like searching a library: your query is "books about transformers," each book's key is its title and topic tags, and the value is the book's actual content. Attention compares your query against all keys to find relevant matches, then retrieves the corresponding values.
How They Work Together
- Query vector: Represents the current position asking "what information do I need?"
- Key vectors: Each input position has a key saying "here's what I contain"
- Value vectors: Each input position has a value holding "here's my actual information"
The attention mechanism computes similarity scores between the query and all keys (those alignment scores you learned about). After softmax normalization, these scores weight the value vectors. The final context vector is a weighted sum of values, where weights come from query-key similarity.
This separation is powerful: keys determine which items are relevant, while values provide what gets retrieved. The query drives the whole process by asking the question.
Key Takeaway: Query, key, and value vectors give attention a flexible retrieval mechanism—the query searches through keys to weight values, creating context-aware representations without the sequential bottleneck of RNNs.