KV Cache Quantization
What You'll Learn
You'll learn how to compress the key-value cache from 16-bit to 8-bit or 4-bit representations to dramatically reduce memory usage while maintaining generation quality.
The Memory Problem
Remember from earlier lessons that KV cache stores keys and values for every token at every layer. For a 7B model with 32 layers serving a 2048-token sequence, that's gigabytes of memory per request! By default, these are stored as 16-bit floats (FP16 or BF16).
The Core Idea
KV Cache Quantization compresses these cached tensors to lower precision formats—typically 8-bit integers (INT8) or even 4-bit. Think of it like compressing a high-resolution photo to JPEG: you lose some fidelity, but the image remains recognizable and usable.
The key insight: keys and values don't need the full precision of model weights. They're intermediate activations used for attention score computation, where small rounding errors have minimal impact on the final output distribution.
How It Works
- Per-tensor or per-channel scaling: Compute scale factors that map the FP16 range to INT8 [-128, 127]
- Quantize on write: When storing new KV pairs during prefill or decode, convert them immediately
- Dequantize on read: When computing attention, convert back to FP16 just-in-time
The memory savings are dramatic: INT8 gives 2× reduction, INT4 gives 4× reduction. For that 7B model example, you've just halved (or quartered) your memory footprint with minimal quality loss—typically less than 1% degradation on benchmarks.
Practical Considerations
The challenge is choosing quantization granularity. Per-tensor quantization is faster but less accurate; per-channel (per-head) quantization preserves more information but adds overhead.
Key Takeaway: KV cache quantization compresses cached attention keys and values to 8-bit or 4-bit formats, cutting memory usage by 2-4× with negligible quality impact—a critical optimization for serving multiple concurrent requests.