Course contentsShow
← Machine Learning and Deep Learning
Lesson 2932 of 3,53863. Model Serving and Inference InfrastructureFree lesson

Service Level Objectives (SLOs) and Budget Allocation

Defining latency SLOs, throughput targets, and making architectural decisions based on business requirements.

Service Level Objectives (SLOs) and Budget Allocation

What you'll learn: How to translate business requirements into concrete latency and throughput targets that guide your model serving architecture decisions.

Why SLOs Matter in ML Serving

An SLO (Service Level Objective) is a quantifiable promise about your system's performance. In model serving, this typically means defining acceptable latency percentiles and minimum throughput requirements. For example: "99% of inference requests must complete within 100ms" or "the system must handle at least 1,000 requests per second."

SLOs bridge the gap between business needs and technical implementation. An e-commerce recommendation system might need sub-50ms latency to avoid cart abandonment, while a nightly batch fraud detection pipeline might tolerate several seconds per prediction.

Defining Your SLOs

Start by asking stakeholders what matters most:

  • Latency SLOs: Often expressed as percentiles (p50, p95, p99). A p99 < 200ms target means 99% of requests must complete under 200ms.
  • Throughput targets: Minimum requests per second the system must sustain during peak load.
  • Availability: Uptime guarantees (e.g., 99.9% availability).

Architecture Decisions Based on SLOs

Your SLOs directly inform technical choices:

  • Tight latency budgets → Smaller batch sizes, faster models, result caching, edge deployment
  • High throughput needs → Dynamic batching, GPU optimization, horizontal scaling
  • Both constraints → Sophisticated request scheduling, multiple model replicas with load balancing

Think of your latency budget like a financial budget: each component (network, preprocessing, model inference, postprocessing) "spends" milliseconds. If your total budget is 100ms and preprocessing takes 30ms, you have only 70ms left for everything else.

Key Takeaway: SLOs transform vague business requirements into precise numerical targets that directly determine your serving infrastructure choices—from batch sizes and caching strategies to hardware selection and deployment topology.