In-Memory Caching with Redis
What You'll Learn
You'll learn how to implement lightning-fast Redis caching for LLM responses with time-based expiration and smart memory management.
Why Redis for LLM Caching?
Redis is an in-memory key-value store that operates at microsecond speeds—perfect for caching expensive LLM responses. Think of it as a super-fast notepad that lives entirely in RAM, letting you retrieve cached answers before the LLM API even receives your request.
Core Implementation Pattern
Redis stores your cache key (remember your carefully designed prompt hash from lesson 955?) alongside the LLM response. When the same prompt arrives again, Redis instantly returns the cached result.
TTL (Time-To-Live) Management: Set expiration times on cached entries to balance freshness with speed. A 24-hour TTL means responses stay cached for a day, then automatically disappear:
import redis
cache = redis.Redis(host='localhost', port=6379, decode_responses=True)
# Cache with 24-hour TTL
cache.setex(
name=cache_key,
time=86400, # seconds
value=llm_response
)
# Retrieve cached response
cached = cache.get(cache_key)
Eviction Policies
When Redis fills up, eviction policies determine what gets deleted:
- allkeys-lru: Removes least-recently-used keys (great for general LLM caching)
- volatile-ttl: Removes keys closest to expiration
- allkeys-lfu: Removes least-frequently-used keys
Configure Redis with maxmemory-policy allkeys-lru to automatically remove stale responses when memory is tight.
Practical Considerations
Set TTLs based on how quickly your content changes. News summaries? 1-hour TTL. Static FAQ responses? 7-day TTL. Use shorter TTLs during model updates to prevent serving outdated cached responses.
Key Takeaway: Redis provides microsecond-latency caching for LLM responses with built-in TTL expiration and intelligent eviction policies, dramatically reducing API costs and improving response times for repeated queries.