Course contentsShow
← AI Engineering
Lesson 1634 of 1,88639. Classical ML Deployment (Non-LLM)Free lesson

Online Serving with REST APIs

Implementing synchronous HTTP endpoints for real-time predictions with sub-second response times.

Online Serving with REST APIs

What You'll Learn

You'll learn how to implement synchronous HTTP endpoints that deliver real-time ML predictions with sub-second response times for immediate user-facing applications.

The Core Concept

REST APIs are the workhorse of real-time ML serving. When a user clicks "Submit" on a form or a mobile app needs instant fraud detection, your model must respond immediately—typically under 100-500 milliseconds. Unlike batch prediction pipelines that process thousands of records overnight, online serving handles one prediction at a time, on-demand.

Think of it like a restaurant: batch inference is meal prep for the week, while REST API serving is a chef cooking to order. Each request arrives independently, your API deserves the input, calls your loaded model, and returns a prediction—all before the user notices any delay.

Implementation Essentials

Your REST endpoint exposes a route (e.g., /predict) that accepts POST requests with JSON payloads containing feature values. The server:

  1. Validates the incoming request schema
  2. Transforms raw inputs using your serialized preprocessing pipeline
  3. Invokes the model's prediction method
  4. Formats the output as JSON with scores or labels
  5. Returns the response with appropriate status codes

Critical considerations:

  • Model warm-up: Load models into memory at startup, not per-request
  • Synchronous blocking: The client waits for the response—no queueing
  • Timeout handling: Set strict deadlines to prevent cascading failures
  • Health checks: Implement /health endpoints for monitoring

This pattern works beautifully for low-to-medium traffic where predictions must be instantaneous—chatbots, recommendation widgets, credit approvals, or dynamic pricing.

Key Takeaway:
REST APIs enable synchronous, sub-second predictions by keeping models hot in memory and processing single requests on-demand, making them ideal for user-facing real-time applications.