Implementing Adaptive Sampling

In this lesson, you will learn how to implement adaptive sampling to dynamically adjust your trace ingestion rates based on real-time system load, ensuring your observability budget remains balanced without sacrificing critical insights during traffic spikes.

The Problem: Fixed vs. Adaptive Sampling

In high-throughput Node.js applications, capturing every single trace creates significant CPU and network overhead. While fixed sampling (e.g., recording 5% of all requests) is simple, it often fails during load spikes—either by flooding your observability backend with redundant data or by dropping traces exactly when you need visibility the most.

Adaptive sampling solves this by treating the sampling rate as a control loop variable. When system CPU usage or event-loop lag increases, the sampler automatically reduces the ingestion rate to preserve application performance. When the system is idle, it increases the rate to provide higher fidelity.

Implementing the Feedback Loop

To implement this, you need a periodic background task that monitors a resource metric (like process.cpuUsage()) and updates your tracer's sampler state.

const { TraceIdRatioBasedSampler } = require('@opentelemetry/sdk-trace-base');

let currentSampler = new TraceIdRatioBasedSampler(0.1); // Default 10%

// Periodic adjustment task
setInterval(() => {
  const load = getSystemLoad(); // Custom function checking CPU/Event Loop
  
  if (load > 0.8) {
    // High load: Throttle down to 1% to save resources
    currentSampler = new TraceIdRatioBasedSampler(0.01);
  } else if (load < 0.2) {
    // Low load: Increase to 25% for better visibility
    currentSampler = new TraceIdRatioBasedSampler(0.25);
  }
}, 5000);

// Use currentSampler in your TracerProvider configuration

Strategic Considerations

When implementing this, avoid making the adjustments too jittery. Use an exponential moving average for your metrics so that temporary micro-bursts don't cause the sampling rate to oscillate wildly. Additionally, ensure that your sampling decision is logged or tagged as a metadata attribute on your spans, so downstream analysis tools understand that the sampling rate was dynamic, preventing skewed statistical calculations in your observability platform.

Key Takeaway: Adaptive sampling creates a self-regulating observability pipeline that protects application performance during high load while maximizing data density during quiet periods.