Per-Request Token Tracking
What you'll learn: How to instrument your application to capture and record token usage for every individual API call.
The Core Problem
You've learned why token usage matters and how to count tokens—but in production, thousands of requests flow through your system daily. Without tracking each request individually, you can't identify which prompts, users, or features are driving costs, nor can you detect anomalies or optimize specific pathways.
What Is Per-Request Tracking?
Per-request token tracking means capturing input tokens, output tokens, and total cost for every single LLM API call your application makes. Think of it like itemized billing on your phone plan—instead of just seeing the monthly total, you see every call, its duration, and its cost.
Implementation Approach
Wrap your API calls with instrumentation that:
- Captures metadata before the call (timestamp, user ID, prompt template, model)
- Records token counts from the API response
- Logs or stores this data for analysis
def track_llm_call(prompt, user_id, model="gpt-4"):
start_time = time.time()
response = openai.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}]
)
# Extract token usage from response
usage = response.usage
cost = calculate_cost(usage.prompt_tokens,
usage.completion_tokens, model)
# Log per-request data
log_token_usage({
"user_id": user_id,
"model": model,
"prompt_tokens": usage.prompt_tokens,
"completion_tokens": usage.completion_tokens,
"total_cost": cost,
"latency": time.time() - start_time
})
return response
This granular data enables cost attribution by user, feature-level optimization, and anomaly detection (like a runaway prompt consuming 10x normal tokens).
Key Takeaway: Per-request token tracking transforms opaque aggregate costs into actionable, line-item insights that drive targeted optimization and cost accountability.