Cache Key Design for Prompts
What you'll learn: How to create robust cache keys that reliably identify when LLM requests can reuse previous responses without missing important variations.
Why Cache Keys Are Critical
A cache key is like a library catalog card—it must uniquely identify what you're looking for. In LLM applications, your cache key determines whether you get a fresh API call (slow, expensive) or a cached response (fast, cheap). Get it wrong, and you'll either miss cache hits or serve stale results to users who changed something important.
What Goes Into a Cache Key
A well-designed cache key includes:
- The prompt itself (exact text or template with filled parameters)
- Model parameters (
temperature,max_tokens,top_p, etc.) - Model version (like
gpt-4-turbo-2024-04-09vsgpt-4-turbo-2024-11-20) - System messages or instructions that shape behavior
Think of it like a recipe: changing the ingredients (prompt), cooking temperature (parameters), or chef (model version) produces different dishes—even if the recipe name stays the same.
Handling Variable Inputs
The tricky part is normalizing variations that don't matter. For example:
- Extra whitespace shouldn't create different cache keys
- Parameter order in templates may not matter
- Default parameter values should match explicit ones
You might hash a normalized representation: trim whitespace, sort parameters, substitute template variables in a consistent order. This ensures "Summarize this: {text}" with text="Hello" always produces the same key.
Versioning Your Keys
Include a cache version identifier so you can invalidate old entries when you change your key generation logic or deploy major prompt updates.
Key Takeaway: Effective cache keys balance precision (capturing all meaningful differences) with robustness (ignoring irrelevant variations), requiring careful consideration of prompt content, model parameters, versions, and normalization strategies.