Log Aggregation for Background Workers

In this lesson, you will learn how to unify logs from multiple background worker instances into a single searchable stream to simplify debugging distributed workflows.

The Challenge of Distributed Logs

When you scale your background workers across multiple containers or servers, logs become fragmented. If a workflow fails, you might find a partial error trace on one worker node, while the triggering event context sits on another. SSH-ing into individual instances to cat log files is unsustainable for complex, asynchronous systems.

Centralized Logging Architecture

Log aggregation solves this by decoupling log generation from log storage. Instead of writing to the local filesystem, your Node.js workers ship logs to a centralized ingestion service (like ELK, Splunk, or Datadog).

To ensure logs are useful, you must include a correlation_id in every log entry. This ID should be passed through the entire lifecycle of a background task, allowing you to filter logs across all workers to see the full journey of a single workflow instance.

Implementation Pattern

Using a library like winston or pino, you can configure a transport that pushes logs over HTTP or UDP, ensuring your application code remains clean.

const logger = pino({
  formatters: {
    // Inject the correlationId automatically into all logs
    bindings: (bindings) => ({ workflowId: currentTaskContext.id })
  },
  transport: {
    target: 'pino-http-transport', // Sends logs to your aggregator
    options: { destination: 'https://logs.your-system.com' }
  }
});

// Within your background worker
async function processTask(task) {
  logger.info({ event: 'task_started' }, 'Worker received task');
  try {
    // ... logic
  } catch (err) {
    logger.error({ err, step: 'database_update' }, 'Workflow failed at step');
  }
}

Why This Matters

Without aggregation, your visibility into your system is limited to a "local view." By centralizing logs, you transform thousands of disconnected entries into a searchable timeline. This allows you to perform cross-worker analysis, such as identifying if a specific queue partition is consistently failing or if latency is only occurring on specific nodes.

Key Takeaway: Never rely on local files for worker logs; treat logs as a stream of events that must be tagged with a workflow-wide correlation ID and shipped to a central repository for instant observability.