Luong Attention Variants
What you'll learn: How Luong attention offers different flavors of attention mechanisms with varying computational costs and focus strategies.
The Core Idea
After Bahdanau introduced attention, Minh-Thang Luong proposed simplified variants that differ in when they compute attention and how they calculate alignment scores. While Bahdanau attention computes attention before generating the decoder's hidden state, Luong attention computes it after.
Two Main Variants
Global Attention
Global attention considers all encoder hidden states when computing attention weights at each decoding step. It's like scanning every word in the source sentence every time you generate a new word in translation. This is comprehensive but computationally expensive for long sequences.
Local Attention
Local attention focuses on a window of encoder states around a predicted position. Imagine predicting where in the source sentence you should focus, then only looking at nearby words. This reduces computation while maintaining reasonable alignment quality—a clever trade-off between speed and accuracy.
Alignment Score Functions
Luong also proposed three ways to compute alignment scores:
- Dot: Simply the dot product between decoder and encoder states (fastest)
- General: Uses a learned weight matrix between states (more flexible)
- Concat: Similar to Bahdanau's approach (most expressive)
These offer different balances between model capacity and computational efficiency.
The Key Difference
The timing matters: Luong computes the decoder hidden state first, then uses it to compute attention. This makes the implementation simpler and slightly faster than Bahdanau's approach.
Key Takeaway: Luong attention variants provide architectural choices—global vs. local attention and different alignment functions—that let you trade off between computational cost and model expressiveness based on your task requirements.