Weighted Sum of Values: Computing Attention Output
What you'll learn: How attention mechanisms combine attention weights with value vectors to create rich, context-aware representations.
The Final Assembly Step
You've computed attention scores, scaled them, and applied softmax to get attention weights—numbers between 0 and 1 that sum to 1. Now comes the payoff: using those weights to build your output.
The attention output is simply a weighted sum of the value vectors, where each value is multiplied by its corresponding attention weight. This is where the magic happens: tokens that received high attention weights contribute more to the final representation, while less relevant tokens contribute less.
Plain English Example
Imagine you're making a smoothie based on ingredient preferences. You have values (strawberries, bananas, spinach) and weights (40%, 50%, 10%). Your final smoothie flavor is:
output = 0.4 × strawberry + 0.5 × banana + 0.1 × spinach
The attention output works identically: each value vector gets "blended" according to its attention weight.
The Mathematical Operation
Given attention weights (from softmax) and value vectors:
attention_output = Σ (attention_weight[i] × value[i])
For each query, you multiply every value vector by its attention weight and sum them all up. The result is a single vector that captures relevant information from across the entire sequence, weighted by importance.
This context-aware representation is what makes attention so powerful—it dynamically focuses on what matters for each query.
Key Takeaway: The attention output is a weighted sum of value vectors, where attention weights determine each value's contribution, producing a context-aware representation that emphasizes relevant information.