Course contentsShow
← Machine Learning and Deep Learning
Lesson 1050 of 3,53824. The Transformer ArchitectureFree lesson

Attention as a Weighted Sum: The Core Idea

Learn how attention computes context as a weighted combination of all input representations.

Attention as a Weighted Sum: The Core Idea

What You'll Learn

You'll understand how attention mechanisms create context by computing a weighted combination of input representations.

The Mathematical Heart of Attention

At its core, attention is remarkably simple: it's just a weighted sum. Instead of forcing all information through a single fixed-length vector (like the context bottleneck in basic encoder-decoder models), attention computes a custom summary for each step by mixing together all available inputs.

How It Works

Think of attention like asking multiple experts for advice, but weighing each expert's opinion differently based on relevance:

  1. You have multiple inputs (encoder hidden states, word embeddings, etc.)
  2. You compute weights (attention weights) that determine how important each input is right now
  3. You multiply each input by its weight and sum everything together

Mathematically, if you have input representations h₁, h₂, ..., hₙ and attention weights α₁, α₂, ..., αₙ, the context vector is:

context = α₁h₁ + α₂h₂ + ... + αₙhₙ

The weights αᵢ always sum to 1 (thanks to softmax normalization), so you're computing a weighted average of all inputs.

Why This Matters

Instead of losing information in a fixed bottleneck, attention selectively focuses on different parts of the input at different times. When translating "The cat sat on the mat" to French, the decoder can look back at "cat" when generating "chat," and at "mat" when generating "tapis" — all through different weighted combinations of the same encoder outputs.

Key Takeaway: Attention computes a context vector as a weighted sum of all input representations, where the weights determine which inputs matter most at each step — enabling dynamic, selective focus instead of fixed compression.