Course contentsShow
← Machine Learning and Deep Learning
Lesson 1039 of 3,53823. Recurrent Neural NetworksFree lesson

Attention Score Computation

Computing alignment scores between decoder state and all encoder hidden states using a scoring function.

Attention Score Computation

What you'll learn: How to calculate alignment scores that determine which encoder hidden states are most relevant for the current decoder step.

The Core Problem

Remember how we discovered that a fixed-length context vector can't capture all the nuances of long sequences? Attention solves this by letting the decoder "look" at all encoder hidden states at each decoding step. But which encoder states matter most? That's where attention scores come in.

What Are Attention Scores?

Attention scores are numerical values that measure how relevant each encoder hidden state is to the current decoder state. Think of it like a spotlight that can shine brighter on certain words in the input sequence depending on what you're currently generating.

The Scoring Function

At each decoder timestep, we compute a score between:

  • The current decoder hidden state (where we are now in generation)
  • Each encoder hidden state (every position in the input sequence)

Common scoring functions include:

  1. Dot product: Simply multiply decoder state with each encoder state
  2. General/multiplicative: Use a learned weight matrix between them
  3. Additive/concat: Concatenate states, pass through a small network

For example, if your encoder processed 10 words, you compute 10 scores — one measuring relevance between the decoder's current state and each of the 10 encoder positions.

These raw scores will later be normalized (using softmax) to create attention weights, but the scoring function is the crucial first step that determines which parts of the input deserve focus.

Key Takeaway: Attention scores quantify the relevance between the decoder's current state and each encoder hidden state using a learned scoring function, forming the foundation for dynamic context selection.