Measuring Inference Performance
What you'll learn: How to measure and understand the key performance metrics that matter when running AI models in production.
When you deploy a model, "it works" isn't enough—you need to know how well it works. Just like measuring a car's performance involves more than just "does it start," measuring inference performance requires tracking several specific metrics.
The Four Key Metrics
Latency is the total time from when you send a request to when you receive the complete response. Think of it as the end-to-end journey time. For a chatbot, this is how long a user waits from hitting "send" to seeing the full answer.
Throughput measures how many requests your system can handle simultaneously or in a given time period (like requests per second). A model might have good latency for one user but struggle when handling 100 concurrent requests.
Tokens per second tells you how fast the model generates output. Since language models produce text token by token, this reveals the actual generation speed. A model generating 20 tokens/second feels much snappier than one generating 5 tokens/second.
Time to first token (TTFT) measures how long before the model starts responding. This is crucial for streaming responses—users perceive a system as "faster" when it starts responding quickly, even if total latency is the same. It's the difference between seeing "typing..." immediately versus staring at a blank screen.
Why All Four Matter
You might have a model with good overall latency but terrible TTFT—users will feel it's laggy. Or great tokens/second but poor throughput—it works great in testing but crashes under real load.
Key Takeaway: Measure latency (total time), throughput (concurrent capacity), tokens/second (generation speed), and time to first token (perceived responsiveness) to truly understand your inference performance.