Benchmarking Local Inference Performance
What you'll learn: How to systematically measure and compare the real-world performance of your locally deployed LLM serving setup.
Why Benchmarking Matters
You've set up Ollama, llama.cpp, vLLM, or another local inference tool—but how do you know if it's actually fast enough for your use case? Benchmarking gives you objective data to answer: "Can this handle my expected load?" and "Which tool or configuration works best for my hardware?"
Think of benchmarking like test-driving cars on your actual commute route. Manufacturer specs are useful, but real performance depends on your roads, traffic, and driving style.
Key Metrics to Measure
Tokens Per Second (Throughput)
How many tokens your system generates per second, both for a single request and across concurrent users. This tells you raw generation speed.
Latency
- Time to First Token (TTFT): How long before the model starts responding
- Inter-Token Latency: Time between subsequent tokens during streaming
- Total Request Time: End-to-end duration
Memory Usage
Peak RAM and VRAM consumption under load. Critical for understanding how many concurrent requests you can handle before running out of memory.
Resource Utilization
CPU/GPU usage percentages help identify bottlenecks—is your GPU maxed out or sitting idle while the CPU struggles?
Comparing Tools for Your Setup
Run the same prompts across different tools (vLLM vs. TGI vs. llama.cpp) with identical hardware. Vary prompt lengths, batch sizes, and concurrent users to find which tool excels at your specific workload pattern—chat completions, document processing, or real-time responses.
Key Takeaway: Benchmarking transforms guesswork into data-driven decisions by measuring tokens/second, latency, and memory usage across your actual hardware and use case, letting you optimize for real production performance.