Logging for Long-running Background Processes

In this lesson, you will learn how to maintain observability in background workers that persist for days or weeks without the benefit of a clean request-response lifecycle.

The Challenge of Persistent Context

Unlike standard web servers where request-id headers provide a natural breadcrumb trail, long-running background processes—such as message queue consumers or cron-like workers—operate in a continuous loop. If a process runs for 500 hours, a single log file will become gargantuan, and identifying which specific task caused a memory leak or a stall becomes impossible if you don't inject synthetic context into every log entry.

Establishing Synthetic Scopes

To make these logs actionable, you must establish a "Task-ID" or "Correlation-ID" for every discrete unit of work the process handles. Because your process doesn't terminate, you cannot rely on process-level logging alone. Instead, use an AsyncLocalStorage wrapper to attach a context identifier to the asynchronous scope of each specific worker execution.

const { AsyncLocalStorage } = require('async_hooks');
const storage = new AsyncLocalStorage();

function runWorkerTask(task) {
  const context = { taskId: task.id, startTime: Date.now() };
  
  storage.run(context, () => {
    logger.info('Processing background job'); // Automatically enriched
    performTask(task);
  });
}

// In your logger setup:
const logger = {
  info: (msg) => {
    const store = storage.getStore();
    const meta = store ? { taskId: store.taskId } : {};
    console.log(JSON.stringify({ msg, ...meta, timestamp: new Date() }));
  }
};

Managing Log Rotation and Pressure

Because background processes never restart, standard log rotation (which often relies on closing a file handle during process exit) can fail if not configured correctly. Ensure your logger is configured to stream to stdout or a dedicated socket rather than writing directly to a flat file. By pushing logs to a collector like Fluentd or Vector, you decouple the process’s uptime from the storage lifecycle. This prevents the "zombie file" issue where the background process maintains an open handle to a deleted log file, silently consuming file descriptors until the worker crashes.

Key Takeaway: Inject a unique synthetic correlation ID into every background unit of work to bridge the gap left by the absence of HTTP request context, and always stream to an external collector to avoid file handle exhaustion in long-lived processes.