Course contentsShow
← System Design
Lesson 1247 of 1,91931. Observability: TracingFree lesson

Trace Querying and Analysis

Master query patterns for finding traces by service, latency, errors, and custom tags to diagnose production issues.

Trace Querying and Analysis

You'll learn: How to efficiently query distributed traces using service names, latency thresholds, error conditions, and custom tags to pinpoint production issues.

Why Query Patterns Matter

Once you've collected traces (via OpenTelemetry, Jaeger, or Zipkin), the real value comes from finding the right traces among millions. Distributed tracing backends store massive amounts of span data—querying effectively is the difference between diagnosing an outage in minutes versus hours.

Core Query Dimensions

By Service Name

Filter traces that pass through specific services: service.name = "payment-api". This narrows down where in your architecture to investigate.

By Latency

Find slow requests: duration > 2s. Critical for identifying performance regressions. You can query minimum duration (slow traces) or maximum duration (faster-than-expected anomalies).

By Error Status

Isolate failures: error = true or http.status_code >= 500. This surfaces traces where something went wrong, often showing exactly which span failed in a multi-service call chain.

By Custom Tags

Leverage span attributes you instrumented: customer.tier = "premium" or feature.flag = "new-checkout". This connects business context to technical behavior.

Combining Filters

The power emerges when combining dimensions:

  • "Show me all traces for checkout-service that took >5s AND returned errors in the last hour"
  • "Find traces tagged with region=us-west where database-service spans exceeded 1s"

Think of it like detective work: Each filter is a clue. Combine them to reconstruct exactly what happened during an incident.

Analysis Techniques

Use trace results to build service dependency graphs, identify critical path bottlenecks, or compare latency distributions before/after deployments. Query results often reveal cascading failures or unexpected service interactions.

Key Takeaway: Mastering trace queries by service, latency, errors, and custom tags transforms raw trace data into actionable production insights, enabling rapid root cause analysis.