Course contentsShow
← Machine Learning and Deep Learning
Lesson 1687 of 3,53836. LLM Inference OptimizationFree lesson

Chunked Prefill for Long Contexts

Processing long prompts in chunks to avoid OOM during prefill while maintaining exact attention semantics.

Chunked Prefill for Long Contexts

What you'll learn: How to process extremely long prompts without running out of GPU memory by splitting the prefill phase into manageable chunks.

The Problem with Long Prompts

When you feed a 100K-token prompt into an LLM, the prefill phase must compute attention scores for all tokens at once. This creates a massive memory spike—the attention matrix alone becomes enormous. For long contexts, this can exceed GPU memory and crash with an out-of-memory (OOM) error, even though the subsequent decode phase (generating one token at a time) would fit comfortably.

How Chunked Prefill Works

Instead of processing all input tokens simultaneously, chunked prefill divides the long prompt into smaller segments (chunks) and processes them sequentially:

  1. Split the input: Break your 100K-token prompt into, say, 10 chunks of 10K tokens each
  2. Process each chunk: Compute attention and KV cache entries for one chunk at a time
  3. Accumulate KV cache: Each chunk's keys and values are stored in the KV cache
  4. Maintain causality: Earlier chunks attend only to themselves; later chunks attend to all previous chunks

Think of it like reading a long book chapter-by-chapter instead of trying to hold every word in your mind at once—you still understand the whole story, but your working memory load stays manageable.

Why It Works

The key insight: attention semantics remain exact. The final KV cache is identical to non-chunked prefill because attention is computed correctly across all prior positions. You're just paying the memory cost gradually rather than all at once.

Key Takeaway: Chunked prefill enables processing ultra-long prompts by splitting the prefill phase into sequential chunks, preventing OOM errors while preserving exact attention behavior and building the complete KV cache incrementally.