Course contentsShow
← Machine Learning and Deep Learning
Lesson 2969 of 3,53865. LLM Inference EnginesFree lesson

The Problem: KV Cache Memory Bottleneck

Understanding how key-value cache memory fragmentation and waste limits LLM serving throughput and batch sizes.

The Problem: KV Cache Memory Bottleneck

You'll learn why the key-value (KV) cache in transformer-based LLMs creates memory bottlenecks that limit serving throughput and batch sizes.

What is the KV Cache?

During autoregressive generation (like in GPT or LLaMA), transformers compute attention over all previous tokens. Instead of recalculating these attention keys and values for every new token, models store them in a KV cache—a memory buffer holding the key and value tensors from each layer for all previously generated tokens.

The Memory Problem

While the KV cache dramatically speeds up generation, it creates a serious memory bottleneck:

  1. Massive per-request memory: For a 7B parameter model with 32 layers, a single 2048-token sequence can require ~1GB of KV cache memory alone
  2. Memory fragmentation: Different requests have different sequence lengths. Allocating fixed-size buffers wastes memory on short sequences; dynamic allocation fragments GPU memory
  3. Wasted capacity: Pre-allocated memory for maximum sequence length sits unused when requests are shorter, preventing additional requests from being batched

The Throughput Impact

Think of it like a parking lot: if you reserve huge spaces for every potential visitor (max sequence length) but most arrive in compact cars (shorter sequences), you're wasting spots and turning away customers who could fit.

This waste directly limits:

  • Batch size: Fewer concurrent requests fit in GPU memory
  • Throughput: Lower batches = underutilized GPU compute
  • Cost efficiency: Expensive hardware sits idle while memory fills with sparse data

Key Takeaway: KV cache memory fragmentation and over-allocation create a critical bottleneck in LLM serving, limiting how many requests you can process simultaneously and often wasting 50%+ of available memory.