Course contentsShow
← AI Engineering
Lesson 1232 of 1,88630. Monitoring and ObservabilityFree lesson

Request-Level Instrumentation

Track individual request metrics including prompt length, completion tokens, total latency, and model parameters used.

Request-Level Instrumentation

What You'll Learn

You'll learn how to capture detailed metrics for every individual LLM request to understand cost, performance, and behavior at the most granular level.

The Core Idea

Request-level instrumentation means recording specific measurements for each API call to your LLM. Instead of just knowing your system handled "1000 requests today," you'll know exactly what happened in request #847: how many tokens were in the prompt, how many were generated, how long it took, which model was used, and more.

Think of it like a receipt for every transaction. When you buy groceries, the receipt doesn't just say "you shopped today"—it lists every item, its price, the timestamp, and the total. Request-level instrumentation does the same for your LLM calls.

What to Track Per Request

At minimum, instrument these key metrics:

  • Prompt token count: How many tokens you sent to the model
  • Completion token count: How many tokens the model generated
  • Total latency: End-to-end time from request to response
  • Model identifier: Which model handled this request (gpt-4, claude-3-opus, etc.)
  • Request timestamp: When the call occurred
  • Status/errors: Success or failure indicators

Why This Matters

With request-level data, you can answer critical questions: Which requests are unexpectedly expensive? Did that timeout happen because of a huge prompt? Is a specific user sending 10x more tokens than others? Are errors correlated with specific models or prompt sizes?

This granular data feeds into your aggregated metrics (from the previous lesson) but also enables debugging individual failures and identifying outliers.

Key Takeaway: Request-level instrumentation captures the complete "receipt" for every LLM call—prompt size, completion length, latency, and model used—enabling precise cost attribution, performance debugging, and usage pattern analysis.