Bahdanau Attention Mechanism
What you'll learn: How Bahdanau attention computes alignment scores before generating each decoder output, enabling dynamic focus on relevant encoder states.
The Core Innovation
Bahdanau attention (also called "additive attention") was the first widely-adopted attention mechanism for sequence-to-sequence models. Unlike the fixed context vector approach, it computes fresh alignment scores at every decoder step, allowing the model to dynamically decide which parts of the input sequence matter most right now.
How It Works
At each decoder timestep, Bahdanau attention:
- Computes alignment scores between the current decoder hidden state and all encoder hidden states using an additive scoring function
- Normalizes these scores with softmax to get attention weights
- Creates a context vector as a weighted sum of encoder states
- Uses this context alongside the decoder's previous hidden state to generate the current output
The key difference from what you've seen before: the alignment computation happens before the decoder produces its output, not after. The decoder's current state queries the encoder states to ask "what should I focus on now?"
Why "Additive"?
The alignment score function combines decoder and encoder states through concatenation and a learned transformation (typically a feedforward layer with tanh activation), then produces a scalar score. This additive combination distinguishes it from other attention variants you might encounter later.
Key Takeaway: Bahdanau attention dynamically computes which encoder states to focus on at each decoder step using an additive alignment function, replacing the bottleneck of fixed context vectors with flexible, step-specific attention.