Course contentsShow
← Machine Learning and Deep Learning
Lesson 1664 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Landmark Attention and Memory Tokens

Special tokens that compress information from distant context for efficient long-range dependencies.

Landmark Attention and Memory Tokens

What you'll learn: How special "landmark" tokens can compress and represent distant context, enabling transformers to handle longer sequences efficiently.

The Core Idea

As context windows grow to thousands or millions of tokens, computing full attention becomes prohibitively expensive. Landmark attention introduces special "memory" or "landmark" tokens that act as compressed summaries of distant portions of the context. Instead of every token attending to every other token, tokens can attend to these landmarks, which in turn represent chunks of the original sequence.

How It Works

Think of landmarks like chapter summaries in a very long book. Rather than re-reading every page to remember what happened 50 chapters ago, you can consult a summary token that "remembers" the key information from that section.

In practice:

  • The input sequence is divided into segments
  • Each segment gets one or more landmark tokens that aggregate information from that segment
  • Tokens attend to nearby tokens directly (like sliding window attention) but use landmark tokens to access distant context
  • Landmarks are computed through pooling, learned projections, or dedicated attention mechanisms

This creates a two-tier hierarchy: local fine-grained attention and global coarse-grained attention via landmarks.

Benefits

Landmark attention reduces the quadratic attention complexity by limiting full attention to local neighborhoods while maintaining long-range dependencies through a smaller set of memory tokens. It's particularly effective when combined with techniques like sliding window attention and sparse attention patterns.

Key Takeaway: Landmark tokens compress distant context into special memory representations, allowing transformers to maintain long-range dependencies without computing full attention over enormous sequences—trading some granularity for dramatic efficiency gains.