Course contentsShow
← AI Engineering
Lesson 955 of 1,88623. LLM Application Architecture PatternsFree lesson

Cache Key Design for Prompts

Creating effective cache keys from prompts, parameters, and model versions while handling variable inputs.

Cache Key Design for Prompts

What you'll learn: How to create robust cache keys that reliably identify when LLM requests can reuse previous responses without missing important variations.

Why Cache Keys Are Critical

A cache key is like a library catalog card—it must uniquely identify what you're looking for. In LLM applications, your cache key determines whether you get a fresh API call (slow, expensive) or a cached response (fast, cheap). Get it wrong, and you'll either miss cache hits or serve stale results to users who changed something important.

What Goes Into a Cache Key

A well-designed cache key includes:

  • The prompt itself (exact text or template with filled parameters)
  • Model parameters (temperature, max_tokens, top_p, etc.)
  • Model version (like gpt-4-turbo-2024-04-09 vs gpt-4-turbo-2024-11-20)
  • System messages or instructions that shape behavior

Think of it like a recipe: changing the ingredients (prompt), cooking temperature (parameters), or chef (model version) produces different dishes—even if the recipe name stays the same.

Handling Variable Inputs

The tricky part is normalizing variations that don't matter. For example:

  • Extra whitespace shouldn't create different cache keys
  • Parameter order in templates may not matter
  • Default parameter values should match explicit ones

You might hash a normalized representation: trim whitespace, sort parameters, substitute template variables in a consistent order. This ensures "Summarize this: {text}" with text="Hello" always produces the same key.

Versioning Your Keys

Include a cache version identifier so you can invalidate old entries when you change your key generation logic or deploy major prompt updates.

Key Takeaway: Effective cache keys balance precision (capturing all meaningful differences) with robustness (ignoring irrelevant variations), requiring careful consideration of prompt content, model parameters, versions, and normalization strategies.