Course contentsShow
← Machine Learning and Deep Learning
Lesson 2923 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Batch-Aware Caching

Combining caching and batching strategies to serve cached results alongside fresh predictions in the same batch.

Batch-Aware Caching

What you'll learn: How to intelligently combine cached predictions with fresh inference requests within the same batch to maximize throughput while minimizing redundant computation.

The Core Idea

When serving models with dynamic batching, some requests in a batch might already have cached results while others need fresh predictions. Batch-aware caching is the strategy of separating cached from uncached requests, processing only what's necessary, and reassembling the full batch response in the correct order.

Without this approach, you'd either:

  • Skip caching entirely and recompute everything (wasteful)
  • Process cached items one-by-one outside the batch (loses batching efficiency)

How It Works

Think of it like a restaurant kitchen: when orders arrive as a batch, the chef checks which dishes are already prepared and warming. Those go straight to serving while only new dishes get cooked—but everything leaves together at the right table.

The typical flow:

  1. Batch arrives with N requests
  2. Check cache for each request using your cache key design
  3. Partition: cached hits go to one group, misses to another
  4. Process misses through the model using dynamic batching
  5. Merge results back into the original request order
  6. Cache new results for future use
  7. Return complete batch response

Why It Matters

This strategy preserves both benefits: you get the throughput gains of batching for fresh requests and zero-latency responses for cached ones. It's particularly powerful for workloads with temporal locality (repeated similar queries) or when using semantic caching with LLMs.

Key Takeaway: Batch-aware caching intelligently splits incoming batches into cached and uncached subsets, processes only what's needed, and reconstructs responses in order—maximizing both cache hit efficiency and batching throughput.