In this lesson, you will learn how to leverage distributed tracing to pinpoint latency bottlenecks that emerge specifically during partial canary traffic shifts.
When performing a canary release using Argo Rollouts, you are effectively splitting traffic between stable and experimental pods. While basic health probes tell you if the pods are "running," they often fail to capture the subtle performance degradation introduced by a new version. Distributed tracing—using tools like Jaeger or Tempo—provides the visibility needed to see if the new version is slowing down downstream dependencies or suffering from inefficient database queries.
Identifying Bottlenecks in the Trace
During a canary phase, the service mesh (like Istio or Linkerd) or an Ingress controller routes a percentage of traffic (e.g., 10%) to your new v2 pods. By instrumenting your application with OpenTelemetry, every request receives a unique TraceID. When you analyze these traces in your observability platform, you can filter by the version label:
# Example filter for Canary analysis
service.name: "order-service"
version: "v2"
duration: > 500ms
If you notice a spike in latency for v2 spans compared to v1, you can drill down into the trace tree. You might discover that the v2 service is making an extra database call or waiting on a legacy microservice that is struggling under the specific connection handling of the newer code.
Correlating Traces with Traffic Shifts
The power of this approach lies in the correlation between your Argo Rollout status and your trace data. As the Rollout controller shifts weight from 10% to 50%, you should overlay your trace latency heatmaps. If the p99 latency rises proportionally with the traffic weight shift, the distributed trace will explicitly show which span—or "child request"—is responsible for the bottleneck. This allows you to differentiate between a general infrastructure issue and a specific performance regression caused by the deployment logic.
Key Takeaway: Distributed tracing allows you to treat canary shifts as controlled experiments, enabling you to detect and debug hidden latency regressions by isolating the specific spans of the new version before a full-scale rollout.