Course contentsShow
← Machine Learning and Deep Learning
Lesson 1041 of 3,53823. Recurrent Neural NetworksFree lesson

Softmax Normalization and Attention Weights

Converting alignment scores into probability distributions that sum to one over the input sequence.

Softmax Normalization and Attention Weights

What You'll Learn

You'll understand how raw alignment scores are transformed into proper attention weights that represent a probability distribution over the input sequence.

The Problem with Raw Scores

In the previous lesson, you learned that alignment scores measure how relevant each input position is to the current decoder state. However, these raw scores have two issues:

  1. They don't sum to 1 – making them hard to interpret as "how much attention to pay"
  2. They can be any magnitude – some might be large positive, others negative, making comparison difficult

The Softmax Solution

The softmax function solves both problems by converting alignment scores into a probability distribution. Think of it like this: imagine you're a teacher grading three essays with raw scores of 85, 92, and 73. Softmax doesn't just rank them—it tells you what percentage of your attention each deserves based on their relative quality.

Given alignment scores [e₁, e₂, ..., eₙ] for n input positions, softmax computes:

α_i = exp(e_i) / Σⱼ exp(e_j)

The exponential (exp) ensures all values are positive, and dividing by the sum ensures they add up to exactly 1.

Attention Weights

The resulting values α₁, α₂, ..., αₙ are called attention weights. Each weight is between 0 and 1, and they collectively form a probability distribution. A weight of 0.7 means "pay 70% of your attention to this input position."

These weights will then be used to create a weighted combination of the encoder's hidden states—the actual context vector.

Key Takeaway: Softmax normalization transforms alignment scores into attention weights—a valid probability distribution that tells the decoder exactly how to distribute its focus across the input sequence.