Course contentsShow
← AI Engineering
Lesson 1031 of 1,88625. Model Serving and Inference OptimizationFree lesson

Memory Requirements of KV Cache

Calculating KV cache memory usage based on model size, sequence length, batch size, and precision.

Memory Requirements of KV Cache

What You'll Learn

You'll learn how to calculate exactly how much memory the KV cache consumes based on your model's configuration and serving parameters.

The Memory Footprint Formula

The KV cache stores key and value tensors for every token, every layer, and every request in your batch. This adds up quickly, and understanding the math helps you plan capacity and avoid out-of-memory errors.

The basic formula:

Memory = 2 × batch_size × sequence_length × num_layers × hidden_size × bytes_per_value

Let's break this down:

  • 2× because we store both keys (K) and values (V)
  • batch_size: how many requests you're processing simultaneously
  • sequence_length: maximum tokens per sequence (input + generated output)
  • num_layers: total transformer layers in your model
  • hidden_size: the dimension of each layer's hidden state
  • bytes_per_value: precision (2 for FP16, 4 for FP32, 1 for INT8)

Real-World Example

For a 7B parameter model like LLaMA-7B serving with:

  • Batch size: 8
  • Max sequence: 2048 tokens
  • 32 layers, 4096 hidden dimensions
  • FP16 precision (2 bytes)

Memory = 2 × 8 × 2048 × 32 × 4096 × 2 = ~17 GB just for KV cache

This is separate from the model weights themselves! With longer sequences or larger batches, the KV cache can exceed your GPU memory before the model does.

Why This Matters

The KV cache grows linearly with sequence length and batch size. A 4× longer sequence means 4× more memory. This is why techniques like prefix caching and paged attention exist—to manage this explosive growth efficiently.

Key Takeaway: KV cache memory scales with batch size, sequence length, layers, and hidden dimensions—calculate it early to avoid production surprises and size your infrastructure appropriately.