Course contentsShow
← AI Engineering
Lesson 1060 of 1,88626. Self-Hosted LLM DeploymentFree lesson

Benchmarking Local Inference Performance

Measuring tokens per second, latency, memory usage, and comparing tools for your hardware and use case.

Benchmarking Local Inference Performance

What you'll learn: How to systematically measure and compare the real-world performance of your locally deployed LLM serving setup.

Why Benchmarking Matters

You've set up Ollama, llama.cpp, vLLM, or another local inference tool—but how do you know if it's actually fast enough for your use case? Benchmarking gives you objective data to answer: "Can this handle my expected load?" and "Which tool or configuration works best for my hardware?"

Think of benchmarking like test-driving cars on your actual commute route. Manufacturer specs are useful, but real performance depends on your roads, traffic, and driving style.

Key Metrics to Measure

Tokens Per Second (Throughput)

How many tokens your system generates per second, both for a single request and across concurrent users. This tells you raw generation speed.

Latency

  • Time to First Token (TTFT): How long before the model starts responding
  • Inter-Token Latency: Time between subsequent tokens during streaming
  • Total Request Time: End-to-end duration

Memory Usage

Peak RAM and VRAM consumption under load. Critical for understanding how many concurrent requests you can handle before running out of memory.

Resource Utilization

CPU/GPU usage percentages help identify bottlenecks—is your GPU maxed out or sitting idle while the CPU struggles?

Comparing Tools for Your Setup

Run the same prompts across different tools (vLLM vs. TGI vs. llama.cpp) with identical hardware. Vary prompt lengths, batch sizes, and concurrent users to find which tool excels at your specific workload pattern—chat completions, document processing, or real-time responses.

Key Takeaway: Benchmarking transforms guesswork into data-driven decisions by measuring tokens/second, latency, and memory usage across your actual hardware and use case, letting you optimize for real production performance.