Course contentsShow
← AI Engineering
Lesson 956 of 1,88623. LLM Application Architecture PatternsFree lesson

In-Memory Caching with Redis

Implementing fast Redis-based caching for LLM responses with TTL management and eviction policies.

In-Memory Caching with Redis

What You'll Learn

You'll learn how to implement lightning-fast Redis caching for LLM responses with time-based expiration and smart memory management.

Why Redis for LLM Caching?

Redis is an in-memory key-value store that operates at microsecond speeds—perfect for caching expensive LLM responses. Think of it as a super-fast notepad that lives entirely in RAM, letting you retrieve cached answers before the LLM API even receives your request.

Core Implementation Pattern

Redis stores your cache key (remember your carefully designed prompt hash from lesson 955?) alongside the LLM response. When the same prompt arrives again, Redis instantly returns the cached result.

TTL (Time-To-Live) Management: Set expiration times on cached entries to balance freshness with speed. A 24-hour TTL means responses stay cached for a day, then automatically disappear:

import redis

cache = redis.Redis(host='localhost', port=6379, decode_responses=True)

# Cache with 24-hour TTL
cache.setex(
    name=cache_key,
    time=86400,  # seconds
    value=llm_response
)

# Retrieve cached response
cached = cache.get(cache_key)

Eviction Policies

When Redis fills up, eviction policies determine what gets deleted:

  • allkeys-lru: Removes least-recently-used keys (great for general LLM caching)
  • volatile-ttl: Removes keys closest to expiration
  • allkeys-lfu: Removes least-frequently-used keys

Configure Redis with maxmemory-policy allkeys-lru to automatically remove stale responses when memory is tight.

Practical Considerations

Set TTLs based on how quickly your content changes. News summaries? 1-hour TTL. Static FAQ responses? 7-day TTL. Use shorter TTLs during model updates to prevent serving outdated cached responses.

Key Takeaway: Redis provides microsecond-latency caching for LLM responses with built-in TTL expiration and intelligent eviction policies, dramatically reducing API costs and improving response times for repeated queries.