Course contentsShow
← Machine Learning and Deep Learning
Lesson 1045 of 3,53823. Recurrent Neural NetworksFree lesson

Luong Attention Variants

Global and local attention mechanisms with different alignment strategies and computational trade-offs.

Luong Attention Variants

What you'll learn: How Luong attention offers different flavors of attention mechanisms with varying computational costs and focus strategies.

The Core Idea

After Bahdanau introduced attention, Minh-Thang Luong proposed simplified variants that differ in when they compute attention and how they calculate alignment scores. While Bahdanau attention computes attention before generating the decoder's hidden state, Luong attention computes it after.

Two Main Variants

Global Attention

Global attention considers all encoder hidden states when computing attention weights at each decoding step. It's like scanning every word in the source sentence every time you generate a new word in translation. This is comprehensive but computationally expensive for long sequences.

Local Attention

Local attention focuses on a window of encoder states around a predicted position. Imagine predicting where in the source sentence you should focus, then only looking at nearby words. This reduces computation while maintaining reasonable alignment quality—a clever trade-off between speed and accuracy.

Alignment Score Functions

Luong also proposed three ways to compute alignment scores:

  • Dot: Simply the dot product between decoder and encoder states (fastest)
  • General: Uses a learned weight matrix between states (more flexible)
  • Concat: Similar to Bahdanau's approach (most expressive)

These offer different balances between model capacity and computational efficiency.

The Key Difference

The timing matters: Luong computes the decoder hidden state first, then uses it to compute attention. This makes the implementation simpler and slightly faster than Bahdanau's approach.

Key Takeaway: Luong attention variants provide architectural choices—global vs. local attention and different alignment functions—that let you trade off between computational cost and model expressiveness based on your task requirements.