Context Window Fundamentals
What you'll learn: How the context window defines what an LLM can "see" at once, and why its length creates a computational bottleneck.
What Is a Context Window?
The context window (or context length) is the maximum number of tokens a language model can process in a single forward pass. Think of it as the model's "working memory"—everything it can attend to when predicting the next token.
For example, if a model has a 2048-token context window, it can only "see" the most recent 2048 tokens of your conversation or document. Anything beyond that is forgotten or must be truncated.
Why Context Length Matters
Longer context windows enable:
- Processing entire documents or books without splitting
- Maintaining coherent long conversations
- Handling complex tasks requiring distant information
- Reducing the need for retrieval augmentation
But there's a significant trade-off.
The Quadratic Scaling Problem
Standard self-attention mechanisms compute relationships between every pair of tokens in the context. If you have n tokens:
- You create n × n attention scores
- Memory and computation scale as O(n²)
Analogy: Imagine a classroom where every student must shake hands with every other student. With 10 students, that's 45 handshakes. With 100 students? 4,950 handshakes. The problem explodes quadratically.
For a 2K context, you compute ~4 million attention scores. For 32K? Over 1 billion. This quadratic growth makes long contexts prohibitively expensive without architectural innovations.
Context Window vs Tokenization
Your effective context depends on how efficiently your tokenizer represents text. A poor tokenizer might use 2 tokens where a good one uses 1, effectively halving your usable context for the same window size.
Key Takeaway: The context window determines an LLM's "memory span," but standard attention's O(n²) scaling makes long contexts computationally expensive—a fundamental challenge driving modern architectural innovations.