Chunked Prefill for Long Contexts
What you'll learn: How to process extremely long prompts without running out of GPU memory by splitting the prefill phase into manageable chunks.
The Problem with Long Prompts
When you feed a 100K-token prompt into an LLM, the prefill phase must compute attention scores for all tokens at once. This creates a massive memory spike—the attention matrix alone becomes enormous. For long contexts, this can exceed GPU memory and crash with an out-of-memory (OOM) error, even though the subsequent decode phase (generating one token at a time) would fit comfortably.
How Chunked Prefill Works
Instead of processing all input tokens simultaneously, chunked prefill divides the long prompt into smaller segments (chunks) and processes them sequentially:
- Split the input: Break your 100K-token prompt into, say, 10 chunks of 10K tokens each
- Process each chunk: Compute attention and KV cache entries for one chunk at a time
- Accumulate KV cache: Each chunk's keys and values are stored in the KV cache
- Maintain causality: Earlier chunks attend only to themselves; later chunks attend to all previous chunks
Think of it like reading a long book chapter-by-chapter instead of trying to hold every word in your mind at once—you still understand the whole story, but your working memory load stays manageable.
Why It Works
The key insight: attention semantics remain exact. The final KV cache is identical to non-chunked prefill because attention is computed correctly across all prior positions. You're just paying the memory cost gradually rather than all at once.
Key Takeaway: Chunked prefill enables processing ultra-long prompts by splitting the prefill phase into sequential chunks, preventing OOM errors while preserving exact attention behavior and building the complete KV cache incrementally.