Course contentsShow
← AI Engineering
Lesson 1638 of 1,88639. Classical ML Deployment (Non-LLM)Free lesson

Choosing Between Online and Offline

Decision framework based on latency needs, data freshness, scale, and cost to select the right inference mode.

Choosing Between Online and Offline Inference

What you'll learn: You'll master the decision framework for selecting online (real-time) versus offline (batch) inference based on latency, freshness, scale, and cost considerations.

The Core Decision Framework

Now that you understand both batch and real-time patterns, how do you choose? This isn't a technical preference—it's a strategic decision driven by four pillars:

1. Latency Requirements

Does your user wait for the prediction? If humans are in the loop expecting sub-second responses (search ranking, fraud detection at checkout), you need online inference. If results can arrive hours or days later (monthly churn predictions, overnight recommendation updates), batch wins.

2. Data Freshness Needs

How stale can your features be? Real-time credit card fraud needs this transaction's data. A weekly marketing campaign can use yesterday's predictions computed in batch.

3. Scale and Throughput

Batch inference processes millions of records efficiently using optimized batching. Online serving handles sporadic requests with variable load. If you're scoring your entire customer base monthly, batch is dramatically cheaper per prediction.

4. Cost Considerations

Online infrastructure runs 24/7, requires load balancing, and often uses expensive low-latency feature stores. Batch jobs spin up, process, and shut down. A simple rule: batch costs scale with work done, online costs scale with time available.

The Hybrid Reality

Many production systems use both: precompute stable features in batch, serve them via a feature store, and add real-time signals during online inference. Think of Netflix: your historical preferences are batch-computed, but "just watched this episode" triggers real-time updates.

Key Takeaway: Choose offline for high-volume, latency-tolerant workloads where freshness exceeds hours; choose online when users wait or when data changes require immediate response; use hybrid architectures to balance cost and responsiveness.