Course contentsShow
← Machine Learning and Deep Learning
Lesson 1657 of 3,53835. Modern Large Language Models: ArchitectureFree lesson

Sliding Window Attention

Restricting attention to local windows to reduce computation while maintaining some long-range capability.

Sliding Window Attention

What you'll learn: How restricting attention to local windows dramatically reduces computational cost while preserving essential long-range modeling capability in large language models.

The Computational Challenge

Full self-attention has quadratic complexity: each token attends to every other token in the context. For a sequence of length n, that's O(n²) operations. When n = 8,192 or 32,768 tokens, this becomes prohibitively expensive in both memory and compute.

The Core Idea

Sliding window attention restricts each token to only attend to a fixed-size window of surrounding tokens—say, the 512 tokens before and after it. This reduces complexity from O(n²) to O(n × w), where w is the window size.

Think of it like reading a book: you don't need to simultaneously remember every word from page 1 while reading page 200. Your "attention" naturally focuses on the current paragraph and maybe the previous few sentences.

How It Works

Instead of computing attention across all positions:

  • Token at position 500 only attends to tokens 0–1012 (if window = 512)
  • Token at position 5000 attends to tokens 4488–5512
  • Each layer maintains local coherence within its window

The clever part: While individual layers see limited context, stacking multiple layers creates longer effective receptive fields. Information propagates through layers, allowing the model to capture dependencies beyond a single window's span.

Practical Impact

Models like Longformer and BigBird combine sliding windows with sparse global tokens to balance efficiency and capability. Some modern LLMs use sliding windows in early layers (for efficiency) and full attention in later layers (for integration).

Key Takeaway: Sliding window attention trades full quadratic attention for linear complexity by restricting each token to local neighbors, enabling much longer context windows while maintaining computational feasibility through multi-layer information propagation.