Course contentsShow
← AI Engineering
Lesson 1632 of 1,88639. Classical ML Deployment (Non-LLM)Free lesson

Latency Requirements and SLAs

Defining acceptable response times and service level agreements that determine online vs offline architecture choices.

Latency Requirements and SLAs

What you'll learn: How to define acceptable response times and service level agreements that fundamentally shape whether you build an online or offline inference architecture.

The Core Concept

Not all predictions are created equal when it comes to timing. A fraud detection system flagging a suspicious credit card transaction needs an answer in milliseconds before the purchase completes. A monthly customer churn prediction model? It can happily run overnight in a batch job.

Latency requirements specify the maximum acceptable delay between a prediction request and its response. Service Level Agreements (SLAs) formalize these expectations as binding commitments—typically expressed as percentiles (e.g., "95% of requests under 100ms") with consequences for violations.

How This Drives Architecture Decisions

Your latency SLA is the single most important factor determining your deployment pattern:

  • Real-time/Online serving (< 100ms): Requires always-on model servers, feature caching, GPU acceleration, and careful optimization of every component in your stack
  • Near-real-time (100ms - 5s): Allows for slightly more complex feature computation and batching strategies
  • Batch/Offline (minutes to hours): Enables cost-effective large-scale processing, complex feature engineering, and ensemble models without time pressure

Think of it like choosing transportation: if you need to cross the country in 3 hours, you must fly—there's no negotiating with a car. Similarly, if your SLA promises 50ms p95 latency, batch inference isn't an option.

Balancing Constraints

Tighter latency requirements often force trade-offs: simpler models, smaller feature sets, higher infrastructure costs, or reduced accuracy. Your SLA should reflect genuine business needs, not arbitrary goals.

Key Takeaway: Latency SLAs are architectural constraints that determine whether you build real-time or batch inference systems—define them based on actual business value, not engineering preferences.