Hybrid Architectures and Precomputation
You'll learn: How to combine batch precomputation with online lookups to optimize for freshness, latency, and compute costs in ML inference.
The Core Idea
Not every prediction needs to be computed on-the-fly, and not every prediction can wait for a nightly batch job. Hybrid architectures split the inference workload: compute what you can ahead of time (batch precomputation), store those results, then serve them instantly via online lookups—only falling back to real-time computation when necessary.
Think of it like a restaurant: you prep ingredients in bulk during slow hours (batch), then quickly assemble dishes when orders come in (online). You don't chop every vegetable to-order, nor do you serve week-old food.
How It Works
- Precompute stable predictions: For entities that change slowly (products, users with historical behavior), run batch predictions daily or hourly and store results in a Feature Store or key-value database
- Online lookup first: When a request arrives, check if a precomputed prediction exists and is fresh enough
- Real-time fallback: For new entities, rapidly changing features, or expired cache entries, invoke the online serving API with real-time feature computation
- Hybrid refresh policies: Configure TTLs (time-to-live) per use-case—recommendations might refresh daily, fraud scores every 5 minutes
Balancing the Tradeoffs
- Latency: Lookups are ~1-10ms vs. 50-500ms for full model inference
- Cost: Batch jobs amortize compute over millions of predictions
- Freshness: Real-time fallback ensures you never serve stale results for critical cases
Key Takeaway: Hybrid architectures with precomputation let you serve most predictions instantly from cache while preserving real-time capability for dynamic scenarios, optimizing the cost-latency-freshness triangle.