Course contentsShow
← AI Engineering
Lesson 1636 of 1,88639. Classical ML Deployment (Non-LLM)Free lesson

Hybrid Architectures and Precomputation

Combining batch precomputation with online lookup to balance freshness, latency, and computational cost.

Hybrid Architectures and Precomputation

You'll learn: How to combine batch precomputation with online lookups to optimize for freshness, latency, and compute costs in ML inference.

The Core Idea

Not every prediction needs to be computed on-the-fly, and not every prediction can wait for a nightly batch job. Hybrid architectures split the inference workload: compute what you can ahead of time (batch precomputation), store those results, then serve them instantly via online lookups—only falling back to real-time computation when necessary.

Think of it like a restaurant: you prep ingredients in bulk during slow hours (batch), then quickly assemble dishes when orders come in (online). You don't chop every vegetable to-order, nor do you serve week-old food.

How It Works

  1. Precompute stable predictions: For entities that change slowly (products, users with historical behavior), run batch predictions daily or hourly and store results in a Feature Store or key-value database
  2. Online lookup first: When a request arrives, check if a precomputed prediction exists and is fresh enough
  3. Real-time fallback: For new entities, rapidly changing features, or expired cache entries, invoke the online serving API with real-time feature computation
  4. Hybrid refresh policies: Configure TTLs (time-to-live) per use-case—recommendations might refresh daily, fraud scores every 5 minutes

Balancing the Tradeoffs

  • Latency: Lookups are ~1-10ms vs. 50-500ms for full model inference
  • Cost: Batch jobs amortize compute over millions of predictions
  • Freshness: Real-time fallback ensures you never serve stale results for critical cases

Key Takeaway: Hybrid architectures with precomputation let you serve most predictions instantly from cache while preserving real-time capability for dynamic scenarios, optimizing the cost-latency-freshness triangle.