Course contentsShow
← Machine Learning and Deep Learning
Lesson 1056 of 3,53824. The Transformer ArchitectureFree lesson

Weighted Sum of Values: Computing Attention Output

Combine attention weights with value vectors to produce context-aware representations.

Weighted Sum of Values: Computing Attention Output

What you'll learn: How attention mechanisms combine attention weights with value vectors to create rich, context-aware representations.

The Final Assembly Step

You've computed attention scores, scaled them, and applied softmax to get attention weights—numbers between 0 and 1 that sum to 1. Now comes the payoff: using those weights to build your output.

The attention output is simply a weighted sum of the value vectors, where each value is multiplied by its corresponding attention weight. This is where the magic happens: tokens that received high attention weights contribute more to the final representation, while less relevant tokens contribute less.

Plain English Example

Imagine you're making a smoothie based on ingredient preferences. You have values (strawberries, bananas, spinach) and weights (40%, 50%, 10%). Your final smoothie flavor is:

output = 0.4 × strawberry + 0.5 × banana + 0.1 × spinach

The attention output works identically: each value vector gets "blended" according to its attention weight.

The Mathematical Operation

Given attention weights (from softmax) and value vectors:

attention_output = Σ (attention_weight[i] × value[i])

For each query, you multiply every value vector by its attention weight and sum them all up. The result is a single vector that captures relevant information from across the entire sequence, weighted by importance.

This context-aware representation is what makes attention so powerful—it dynamically focuses on what matters for each query.

Key Takeaway: The attention output is a weighted sum of value vectors, where attention weights determine each value's contribution, producing a context-aware representation that emphasizes relevant information.