Sampling Strategies for High Traffic

In this lesson, you will learn how to implement probabilistic sampling to maintain visibility into your Node.js services without the unsustainable storage costs of capturing every single request.

The Problem: Trace Explosion

When your service handles thousands of requests per second, recording a full distributed trace for every interaction creates a massive volume of telemetry data. This leads to prohibitive storage costs, high network overhead for your observability backend, and a "needle in a haystack" problem when searching for specific insights. Probabilistic sampling allows us to record a statistically significant subset of requests, ensuring we capture enough data to identify trends and anomalies while discarding the majority of redundant traces.

Implementing Probabilistic Sampling

The core strategy is to make a decision at the start of a request: do we trace this, or do we ignore it?

By using a probability constant (e.g., 0.1 for 10% sampling), we can use a simple random number generator to determine the fate of a trace.

const SAMPLING_RATE = 0.1; // 10%

function shouldSample() {
  return Math.random() < SAMPLING_RATE;
}

// In your middleware or request handler:
function handleRequest(req, res) {
  const isSampled = shouldSample();
  
  if (isSampled) {
    const trace = startTrace('incoming-request');
    // ... attach trace to request context
  }

  try {
    // Process business logic
  } finally {
    if (isSampled) {
      endTrace();
    }
  }
}

Strategic Considerations

While Math.random() works for basic use cases, production-grade systems often use Head-based sampling. This means the decision to sample is made at the entry point of your architecture (e.g., the API Gateway or Load Balancer).

The critical rule here is Context Propagation: once a decision is made to sample a request, that decision must be passed along to all downstream services via tracing headers (like traceparent). If Service A decides to sample, Service B must respect that decision, or your trace will contain "holes," rendering the diagnostic process useless. By propagating the sampling flag, you ensure that if you record 10% of entry requests, you see the full end-to-end flow for that entire 10% sample across your distributed system.

Key Takeaway: Probabilistic sampling drastically reduces telemetry overhead by recording only a representative fraction of traffic, provided that the sampling decision is consistently propagated across all service boundaries.