Course contentsShow
← AI Engineering
Lesson 1143 of 1,88628. Prompt and Chain OptimizationFree lesson

Creating Latency Baselines and Budgets

Set target latency thresholds per component and track variance across runs to detect regressions.

Creating Latency Baselines and Budgets

You'll learn: How to establish target latency thresholds for each component in your LLM pipeline and monitor them systematically to catch performance degradations early.

Why Latency Budgets Matter

Now that you can measure latency across your entire chain, you need to know what's acceptable. A latency budget is like a financial budget—you allocate specific time "costs" to each component (prompt processing, vector search, LLM generation) and ensure you stay within your total target.

Without baselines, you're flying blind. A query that takes 3 seconds today might take 5 seconds next week due to code changes, model updates, or infrastructure drift—but you won't notice until users complain.

Establishing Baselines

Run your pipeline repeatedly under typical conditions and record the median, p95, and p99 latencies for each component. These become your baseline. For example:

  • Prompt processing: 50ms (p95: 80ms)
  • Vector search: 200ms (p95: 350ms)
  • LLM generation: 2000ms (p95: 2800ms)
  • Total budget: 2500ms (p95: 3500ms)

Setting Budgets and Tracking Variance

Assign a threshold to each component—typically 10-20% above your baseline p95. Track variance across runs. If your vector search suddenly spikes from 200ms to 600ms consistently, you've detected a regression. Automate alerts when components exceed their budgets.

Key Takeaway: Latency baselines provide your "normal" performance profile, while budgets set acceptable limits—tracking variance against these budgets lets you detect and fix performance regressions before they impact users.