Securing Distributed Tracing Data

In this lesson, you will learn how to prevent sensitive information from leaking into your distributed tracing backends by implementing robust span attribute sanitization.

The Exposure Risk

When we instrument applications using OpenTelemetry, we often add attributes to our spans to gain context about a request. While tagging a span with a user.id or a tenant.id is helpful, developers often accidentally include sensitive headers like Authorization tokens, Cookie contents, or personally identifiable information (PII) directly into span attributes. Because these traces are frequently exported to third-party SaaS observability platforms, this metadata is effectively "leaked" to an external environment outside your security perimeter.

Sanitizing Exported Data

To secure your telemetry, you must treat span attributes as untrusted data. Instead of attaching raw request objects to spans, use a middleware or a custom processor to redact sensitive keys.

Here is how you can implement a basic sanitization pattern when adding attributes to your spans:

const { SpanKind } = require('@opentelemetry/api');

function addSecureAttribute(span, key, value) {
  const sensitiveKeys = ['authorization', 'cookie', 'set-cookie', 'password'];
  
  if (sensitiveKeys.includes(key.toLowerCase())) {
    span.setAttribute(key, '[REDACTED]');
  } else {
    span.setAttribute(key, value);
  }
}

// Usage in your request handler
const span = tracer.startSpan('process-request');
addSecureAttribute(span, 'http.request.header.authorization', req.headers.authorization);

Implementing Span Processors

For a more architectural approach, you can implement a custom SpanProcessor within your OpenTelemetry SDK. A processor allows you to intercept every span before it is exported to the collector. By iterating through the span's attributes and stripping or masking keys that match your blacklist (like jwt or api_key), you create a "safety valve" that ensures no developer oversight can result in a data breach. This is critical in distributed systems where a single insecure microservice could inadvertently export credentials belonging to a different service.

Key Takeaway: Always sanitize span attributes before export to ensure that PII and security credentials never leave your infrastructure, even when using third-party observability tools.