Course contentsShow
← AI Engineering
Lesson 1177 of 1,88629. Cost OptimizationFree lesson

Per-Request Token Tracking

Implementing instrumentation to capture token counts for every API call in your application.

Per-Request Token Tracking

What you'll learn: How to instrument your application to capture and record token usage for every individual API call.

The Core Problem

You've learned why token usage matters and how to count tokens—but in production, thousands of requests flow through your system daily. Without tracking each request individually, you can't identify which prompts, users, or features are driving costs, nor can you detect anomalies or optimize specific pathways.

What Is Per-Request Tracking?

Per-request token tracking means capturing input tokens, output tokens, and total cost for every single LLM API call your application makes. Think of it like itemized billing on your phone plan—instead of just seeing the monthly total, you see every call, its duration, and its cost.

Implementation Approach

Wrap your API calls with instrumentation that:

  1. Captures metadata before the call (timestamp, user ID, prompt template, model)
  2. Records token counts from the API response
  3. Logs or stores this data for analysis
def track_llm_call(prompt, user_id, model="gpt-4"):
    start_time = time.time()
    
    response = openai.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    
    # Extract token usage from response
    usage = response.usage
    cost = calculate_cost(usage.prompt_tokens, 
                          usage.completion_tokens, model)
    
    # Log per-request data
    log_token_usage({
        "user_id": user_id,
        "model": model,
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "total_cost": cost,
        "latency": time.time() - start_time
    })
    
    return response

This granular data enables cost attribution by user, feature-level optimization, and anomaly detection (like a runaway prompt consuming 10x normal tokens).

Key Takeaway: Per-request token tracking transforms opaque aggregate costs into actionable, line-item insights that drive targeted optimization and cost accountability.