Course contentsShow
← Machine Learning and Deep Learning
Lesson 1065 of 3,53824. The Transformer ArchitectureFree lesson

Attention vs Traditional Sequence Models

Compare attention's parallel processing and global context to RNN's sequential limitations.

Attention vs Traditional Sequence Models

What you'll learn: How attention mechanisms fundamentally differ from RNNs/LSTMs in processing sequential data.

The Sequential Bottleneck

Traditional RNN-based models (including LSTMs and GRUs) process sequences one step at a time, like reading a book word-by-word while trying to remember everything that came before. Each hidden state must compress all previous information, creating a bottleneck—especially for long sequences.

Analogy: Imagine listening to a long story and having to remember every detail in order. By the end, you might forget important points from the beginning because your memory has limited capacity.

Attention's Parallel Processing Advantage

Attention mechanisms, particularly self-attention, can process all positions simultaneously. Instead of sequential computation, attention looks at the entire sequence at once and directly computes relationships between any two positions—no matter how far apart.

Key differences:

  • RNN/LSTM: Must process position 1, then 2, then 3... (serial)
  • Attention: Processes all positions in parallel, computing position 1↔100 as easily as position 1↔2

Global Context Without Compression

With RNNs, information from position 1 must flow through every intermediate state to reach position 100, degrading along the way. Attention allows direct access to any position's information through the attention weights—no compression into a fixed-size vector.

This means:

  • No vanishing gradient issues across long distances
  • Constant-time path length between any two positions (vs. linear in RNNs)
  • Better capture of long-range dependencies

Key Takeaway: Attention replaces RNN's sequential, bottlenecked processing with parallel, direct connections between all sequence positions, enabling global context understanding and efficient training on long sequences.