KV Cache and Provider-Side Caching
What you'll learn: How OpenAI, Anthropic, and other providers use KV caching to reduce your costs and latency by reusing computation from repeated prompt prefixes.
What Is KV Cache?
When an LLM processes text, it builds key-value (KV) pairs for each token to track attention relationships. Recomputing these for identical prompt prefixes wastes GPU cycles and money.
KV cache stores these intermediate computations so that when you send a prompt with a shared prefix (like system instructions), the provider can skip reprocessing those tokens entirely.
Provider-Side Caching
OpenAI's prompt caching (available on GPT-4 and newer) and Anthropic's prefix caching automatically detect when you're sending prompts with identical beginnings. They cache the KV pairs from the common prefix and only process the new tokens.
Think of it like this: Instead of reading an entire book every time you want to discuss a new chapter, you bookmark where the new content starts and pick up from there.
How It Works in Practice
- Automatic detection: Providers identify shared prefixes across your API calls
- Cache duration: Typically 5-60 minutes depending on provider
- Cost savings: 50-90% reduction for cached tokens (check provider pricing)
- Latency gains: Faster time-to-first-token for long system prompts
You don't change your code—just structure prompts with stable prefixes (system instructions, long documents) that stay consistent across requests.
Why This Matters
Long system prompts (few-shot examples, instructions) used to be expensive on every call. With provider-side caching, you pay full price once, then discounted rates for subsequent hits while the cache is warm.
Key Takeaway: Provider-side KV caching automatically shares computation for repeated prompt prefixes, dramatically reducing both cost and latency without requiring code changes—just thoughtful prompt structure.