Course contentsShow
← Machine Learning and Deep Learning
Lesson 1677 of 3,53836. LLM Inference OptimizationFree lesson

Sliding Window Attention

Trading global context for bounded memory by restricting attention to recent tokens.

Sliding Window Attention

What you'll learn: How to drastically reduce memory usage during LLM inference by limiting each token's attention to only its nearest neighbors.

The Core Idea

In standard self-attention, every token attends to all previous tokens in the sequence—if you have 100,000 tokens, the 100,000th token computes attention over 99,999 predecessors. This creates massive KV cache requirements that grow quadratically.

Sliding window attention imposes a hard limit: each token only attends to the previous W tokens (the "window size"), ignoring everything older. If W=4096, token 100,000 only sees tokens 96,001–99,999. The distant past is simply forgotten.

Why This Helps

By capping attention to a fixed window, memory usage becomes bounded and predictable. Your KV cache only needs to store W entries per layer, no matter how long the sequence grows. This enables:

  • Predictable memory footprint
  • Support for arbitrarily long sequences
  • Faster attention computation (fewer keys/values to process)

The Trade-off

You lose global context. A token 10,000 steps back won't directly influence current predictions. However, information can still propagate through the network via multiple layers—deeper layers see increasingly aggregated representations from earlier windows.

Analogy: Instead of remembering your entire conversation history word-for-word, you only recall the last few sentences verbatim but retain general themes through summarization.

Practical Impact

Models like Mistral 7B use sliding window attention (W=4096) successfully, showing that many tasks don't require full global attention. Combined with prefix caching for shared context, sliding windows enable efficient long-document processing.

Key Takeaway: Sliding window attention trades global context for O(1) memory growth per token, enabling efficient inference on sequences far longer than the attention window by relying on multi-layer information flow.