Memory Layout in Traditional LLM Serving
What you'll learn: How traditional LLM inference systems allocate memory for the KV cache and why this approach causes both internal and external fragmentation.
The Standard Memory Allocation Pattern
When serving LLMs, each request needs to store key-value (KV) cache tensors for every generated token. Traditional systems pre-allocate contiguous blocks of memory for each sequence's entire KV cache upfront, based on the maximum possible sequence length.
Think of it like reserving a full parking lot for a single event: you block off 100 spaces even if only 30 cars show up, and those empty spaces can't be used by anyone else.
Internal Fragmentation: The Wasted Space Within
Internal fragmentation occurs because you allocate memory for the maximum sequence length, but most sequences finish earlier. If you allocate 2048 tokens worth of KV cache but the response is only 200 tokens long, you've wasted ~90% of that reserved memory—and it remains locked until the request completes.
External Fragmentation: The Gaps Between Allocations
External fragmentation happens when completed requests free their memory blocks, leaving gaps. A new request needing a large contiguous block might fail even though total free memory is sufficient—it's just scattered across non-adjacent fragments.
Imagine a bookshelf where removed books leave gaps: you might have space for 5 books total, but not enough consecutive shelf space for a 3-book series.
Why Contiguous Memory?
GPUs expect tensors in contiguous memory for efficient computation. Traditional serving systems prioritize simple memory management over utilization, leading to significant waste at scale—sometimes only 20-30% of allocated memory actually stores useful KV data.
Key Takeaway: Traditional LLM serving pre-allocates contiguous memory for maximum sequence length, causing internal fragmentation (unused space within allocations) and external fragmentation (scattered gaps between allocations), severely limiting batch sizes and throughput.