Course contentsShow
← Machine Learning and Deep Learning
Lesson 1038 of 3,53823. Recurrent Neural NetworksFree lesson

The Core Idea Behind Attention

Allow the decoder to focus on different parts of the input sequence at each decoding step.

The Core Idea Behind Attention

What you'll learn: How attention mechanisms let decoders selectively focus on relevant parts of the input sequence instead of relying on a single fixed context vector.

The Problem with Fixed Context

In the previous lesson, you learned that encoder-decoder models compress an entire input sequence into a single fixed-length context vector. This creates a bottleneck: the decoder must extract all necessary information from just this one vector, regardless of what it's currently generating.

Imagine translating a long English sentence into French. When generating the first French word, you probably care most about the first few English words. When generating the last French word, the ending of the English sentence matters most. But with a fixed context vector, the decoder sees the same representation at every step.

Attention: Dynamic Focus

Attention solves this by allowing the decoder to "look back" at the entire input sequence at each decoding step and dynamically choose which parts to focus on.

Think of it like reading a textbook while writing an essay. Instead of memorizing the entire book (fixed context), you keep it open and flip back to relevant sections as you write each paragraph. When discussing Chapter 3's topic, you focus on Chapter 3. When referencing Chapter 7, you attend to those pages.

At each decoding step, attention:

  1. Examines all encoder hidden states (not just the last one)
  2. Computes relevance scores for each encoder state
  3. Creates a weighted combination emphasizing the most relevant parts
  4. Uses this custom context vector for generating the current output

Key Takeaway: Attention mechanisms let decoders dynamically focus on different parts of the input sequence at each generation step, eliminating the fixed-context bottleneck and dramatically improving sequence-to-sequence performance.