KV Cache Memory Requirements
What You'll Learn
You'll learn how to calculate the memory overhead of the key-value cache during LLM inference and understand which factors drive memory consumption.
Understanding Memory Overhead
When generating text with large language models, the KV cache stores attention keys and values from previous tokens to avoid redundant computation. However, this optimization comes with a memory cost that can quickly become the bottleneck in serving LLMs at scale.
The Memory Calculation
The memory required for KV cache depends on four key factors:
Batch size (B): Each request in a batch needs its own cache, multiplying memory linearly.
Sequence length (S): As generation progresses, the cache grows with each new token. A 2,048-token conversation requires twice the memory of a 1,024-token one.
Number of layers (L): Every transformer layer maintains separate key and value caches. A 32-layer model needs 32× more cache memory than a single layer.
Hidden dimension (D): The size of each key/value vector. Larger models with wider layers (e.g., 4,096 vs 2,048) quadruple the per-token memory.
The formula looks like this:
Memory ≈ 2 × B × S × L × D × bytes_per_parameter
The factor of 2 accounts for both keys and values. For FP16 precision, that's 2 bytes per parameter.
Real-World Example
A 7B parameter model with 32 layers, hidden dimension 4,096, serving a batch of 8 requests at sequence length 2,048:
2 × 8 × 2048 × 32 × 4096 × 2 bytes ≈ 8.6 GB
That's just for the cache—before counting the model weights themselves!
Key Takeaway: KV cache memory scales with batch size, sequence length, layer count, and hidden dimensions—making memory management critical for efficient LLM serving, especially with long contexts or large batches.