Handling Trace Spans for Failed Requests

Force trace collection for all error-prone requests to assist in post-mortem analysis.

In this lesson, you will learn how to bypass standard sampling rates to ensure that every failed request is fully captured, providing the granular data needed for post-mortem analysis.

The Problem with Probabilistic Sampling

In high-traffic Node.js applications, we often use "Head-based Sampling" to reduce telemetry costs, where we decide to keep or drop a trace at the very beginning of a request. While this is efficient for healthy traffic, it creates a "blind spot": if an error occurs during a request that was sampled out, you are left with no trace data to diagnose the root cause.

Forcing Trace Collection

To solve this, you need to implement "Tail-based sampling" or, at a minimum, an error-triggered override. The logic is simple: when an error is caught during the request lifecycle, you manually set the span's status to ERROR and ensure the Sampled flag is toggled on before the span is finalized.

In OpenTelemetry, this involves interacting with the span object directly when your error-handling middleware intercepts a failure.

const { SpanStatusCode } = require('@opentelemetry/api');

function handleRequestError(err, span) {
  // 1. Mark the span as failed
  span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
  
  // 2. Record the exception details
  span.recordException(err);

  // 3. Ensure this trace is exported regardless of initial sampling
  // Note: Depending on your exporter, you may need to check 
  // the 'sampled' flag on the SpanContext
  span.setAttribute('sampling.priority', 1); 
  
  span.end();
}

Strategic Implementation

By marking the span with a high priority attribute, you instruct your trace processor or collector (like the OTel Collector) to ignore the standard sampling rules for that specific trace ID. This ensures that even if you are only sampling 5% of your total traffic, your error logs will show 100% of your failures with complete context—including database queries, upstream headers, and internal logic flow.

Key Takeaway: Never lose visibility on critical failures; intercept errors in your middleware to force the retention of traces, ensuring every bug has a corresponding trail of evidence for your post-mortem analysis.