Latency Requirements and SLAs
What you'll learn: How to define acceptable response times and service level agreements that fundamentally shape whether you build an online or offline inference architecture.
The Core Concept
Not all predictions are created equal when it comes to timing. A fraud detection system flagging a suspicious credit card transaction needs an answer in milliseconds before the purchase completes. A monthly customer churn prediction model? It can happily run overnight in a batch job.
Latency requirements specify the maximum acceptable delay between a prediction request and its response. Service Level Agreements (SLAs) formalize these expectations as binding commitments—typically expressed as percentiles (e.g., "95% of requests under 100ms") with consequences for violations.
How This Drives Architecture Decisions
Your latency SLA is the single most important factor determining your deployment pattern:
- Real-time/Online serving (< 100ms): Requires always-on model servers, feature caching, GPU acceleration, and careful optimization of every component in your stack
- Near-real-time (100ms - 5s): Allows for slightly more complex feature computation and batching strategies
- Batch/Offline (minutes to hours): Enables cost-effective large-scale processing, complex feature engineering, and ensemble models without time pressure
Think of it like choosing transportation: if you need to cross the country in 3 hours, you must fly—there's no negotiating with a car. Similarly, if your SLA promises 50ms p95 latency, batch inference isn't an option.
Balancing Constraints
Tighter latency requirements often force trade-offs: simpler models, smaller feature sets, higher infrastructure costs, or reduced accuracy. Your SLA should reflect genuine business needs, not arbitrary goals.
Key Takeaway: Latency SLAs are architectural constraints that determine whether you build real-time or batch inference systems—define them based on actual business value, not engineering preferences.