Computing the Context Vector
What you'll learn: How attention weights combine encoder hidden states into a single, information-rich context vector that guides decoding.
The Weighted Sum Operation
Now that we have attention weights (one normalized score per encoder time step), we use them to create the context vector. This is simply a weighted sum of all encoder hidden states, where each hidden state is multiplied by its corresponding attention weight.
Think of it like a recipe: if you have ingredients (encoder hidden states) and importance percentages (attention weights), you mix them proportionally. A 70% weight on one hidden state means it contributes more "flavor" to your final context vector than a state with only 5%.
Why This Matters
The context vector is dynamic—it changes at every decoder time step based on what the decoder currently needs. Unlike the fixed bottleneck vector we discussed earlier, this context vector can emphasize different parts of the input sequence as decoding progresses.
For example, when translating "The cat sat on the mat" to French:
- When generating "chat" (cat), attention weights focus on the encoder state for "cat"
- When generating "tapis" (mat), weights shift to emphasize "mat"
The resulting context vector for each step is tailored to that moment's translation needs.
The Math
If encoder hidden states are h₁, h₂, ..., hₙ and attention weights are α₁, α₂, ..., αₙ, then:
context = α₁ × h₁ + α₂ × h₂ + ... + αₙ × hₙ
This is a straightforward weighted average, producing a single vector that captures relevant information from across the entire input sequence.
Key Takeaway: The context vector is computed as a weighted sum of encoder hidden states using attention weights, creating a flexible, query-specific summary of the input sequence.