Latency Components in Inference Pipelines
What you'll learn: How to dissect end-to-end inference latency into distinct components so you can identify and optimize bottlenecks.
Understanding Total Latency
When a user requests a prediction from your ML service, the clock starts ticking the moment their request leaves their device and stops when the response arrives back. That total time isn't just "model execution"—it's a sequence of distinct stages, each contributing its own delay.
Think of it like ordering food at a drive-through: you wait in line (queuing), place your order (network transmission), the kitchen preps ingredients (preprocessing), cooks your meal (model execution), plates it nicely (postprocessing), and finally hands it to you (network transmission back). Each step adds time.
The Five Key Components
1. Network Overhead
Time spent transmitting the request to your server and sending the response back. Depends on payload size, network latency, and protocol choice (REST vs gRPC).
2. Preprocessing
Parsing input data, tokenization, image resizing, normalization, feature extraction—everything needed to convert raw input into model-ready tensors.
3. Model Execution
The actual forward pass through your neural network. This is what you trained, but often not the dominant latency contributor in production.
4. Postprocessing
Converting raw model outputs into useful results: applying softmax, decoding text, drawing bounding boxes, formatting JSON responses.
5. Queuing Delays
Time requests spend waiting when your service is at capacity. Related to concurrency limits, batch formation (from dynamic batching), and load patterns.
Why This Matters
You can't optimize what you don't measure. A model that runs in 10ms might deliver 200ms user-facing latency if preprocessing takes 50ms, queuing adds 100ms, and network contributes 40ms. Profiling each component tells you where to focus optimization effort.
Key Takeaway: Total inference latency is the sum of network overhead, preprocessing, model execution, postprocessing, and queuing delays—measure each separately to find your actual bottlenecks.