Trace Querying and Analysis
You'll learn: How to efficiently query distributed traces using service names, latency thresholds, error conditions, and custom tags to pinpoint production issues.
Why Query Patterns Matter
Once you've collected traces (via OpenTelemetry, Jaeger, or Zipkin), the real value comes from finding the right traces among millions. Distributed tracing backends store massive amounts of span data—querying effectively is the difference between diagnosing an outage in minutes versus hours.
Core Query Dimensions
By Service Name
Filter traces that pass through specific services: service.name = "payment-api". This narrows down where in your architecture to investigate.
By Latency
Find slow requests: duration > 2s. Critical for identifying performance regressions. You can query minimum duration (slow traces) or maximum duration (faster-than-expected anomalies).
By Error Status
Isolate failures: error = true or http.status_code >= 500. This surfaces traces where something went wrong, often showing exactly which span failed in a multi-service call chain.
By Custom Tags
Leverage span attributes you instrumented: customer.tier = "premium" or feature.flag = "new-checkout". This connects business context to technical behavior.
Combining Filters
The power emerges when combining dimensions:
- "Show me all traces for
checkout-servicethat took >5s AND returned errors in the last hour" - "Find traces tagged with
region=us-westwheredatabase-servicespans exceeded 1s"
Think of it like detective work: Each filter is a clue. Combine them to reconstruct exactly what happened during an incident.
Analysis Techniques
Use trace results to build service dependency graphs, identify critical path bottlenecks, or compare latency distributions before/after deployments. Query results often reveal cascading failures or unexpected service interactions.
Key Takeaway: Mastering trace queries by service, latency, errors, and custom tags transforms raw trace data into actionable production insights, enabling rapid root cause analysis.