Semantic Caching for LLMs
What you'll learn: How to dramatically reduce LLM inference costs and latency by caching responses based on meaning rather than exact text matches.
The Problem with Traditional Caching
Traditional cache key design uses exact string matching—if a user asks "What's the capital of France?" and later someone asks "What is France's capital city?", you get two cache misses despite the questions being semantically identical. This wastes compute and money, especially with expensive LLM inference.
How Semantic Caching Works
Instead of using the raw prompt as a cache key, semantic caching:
- Embeds the prompt using a lightweight embedding model (like
sentence-transformers) - Compares similarity to previously cached prompt embeddings using cosine similarity or vector search
- Returns the cached response if similarity exceeds a threshold (e.g., 0.95)
Think of it like recognizing that "automobile" and "car" mean the same thing, rather than treating them as completely different words.
The Architecture
You maintain a vector database or in-memory store containing:
- Prompt embeddings (keys)
- Original prompts (for validation)
- Cached LLM responses (values)
- Metadata (timestamp, hit count)
When a new prompt arrives, you embed it, search for similar embeddings, and return the cached response if the similarity is high enough—bypassing the expensive LLM call entirely.
Similarity Threshold Tuning
Setting the threshold too high (e.g., 0.99) misses semantically similar prompts. Too low (e.g., 0.80) risks returning incorrect responses for genuinely different questions. This requires careful tuning based on your domain and acceptable error rate.
Key Takeaway: Semantic caching uses embedding similarity to recognize when prompts mean the same thing, dramatically reducing LLM costs by reusing responses for semantically equivalent questions instead of requiring exact matches.