Alignment Score Functions
What you'll learn: How different mathematical functions measure the relevance between encoder hidden states and the current decoder state in attention mechanisms.
The Core Idea
In the previous lesson, you learned that attention computes scores between decoder and encoder states. But how exactly do we calculate those scores? That's where alignment score functions come in—they're the specific mathematical formulas that measure how well two vectors "align" or match.
Think of it like judging compatibility in dating: different people use different criteria (shared interests, values, chemistry). Similarly, different score functions use different mathematical operations to measure alignment.
Three Common Scoring Mechanisms
Dot Product (Simplest)
The most straightforward approach: multiply corresponding elements and sum them up.
score = decoder_state · encoder_state
This is like asking "how similar are these vectors?" Higher dot products mean more aligned directions.
Additive (Bahdanau Attention)
Uses a learned neural network layer to compute alignment:
score = v^T · tanh(W1 · encoder_state + W2 · decoder_state)
Here, W1, W2, and v are learnable weight matrices. The network learns what makes states relevant to each other—like learning what compatibility factors matter most through experience.
Multiplicative (Luong Attention)
A middle ground that uses a single weight matrix:
score = decoder_state^T · W · encoder_state
This transforms one vector before comparing, allowing more flexibility than pure dot product but simpler than additive.
Why Multiple Options?
Different tasks benefit from different complexity levels. Dot product is fast but assumes states are already in comparable spaces. Additive is most expressive but slower. Multiplicative balances both.
Key Takeaway: Alignment score functions are the mathematical heart of attention—they determine how we measure relevance between sequence positions, with dot product, additive, and multiplicative being the three fundamental approaches.