Batch Processing Jobs

In this lesson, you will learn how to optimize your system's performance by grouping individual tasks into single, larger operations to reduce database strain and improve throughput.

The Problem with Granularity

When you process jobs one by one, your background worker spends a significant amount of time performing "overhead" operations—such as opening a database transaction, establishing a network connection, or writing a log entry. If you have 1,000 tasks that each involve writing a status update to a database, performing 1,000 individual INSERT or UPDATE queries creates massive write contention and forces your database to manage 1,000 separate transaction logs.

The Batching Strategy

Batch processing allows you to aggregate these granular tasks into "chunks." Instead of processing the queue item-by-item, your worker waits until it has collected a specific number of jobs (e.g., 100) or until a specific timeframe has passed (e.g., 5 seconds).

By grouping them, you turn 1,000 individual network round-trips into 10 bulk operations. This significantly reduces the overhead on your database and speeds up the overall execution time of your job queue.

Example: Bulk Database Inserts

Instead of executing a query every time a job is picked up, you can buffer them in memory:

const taskBuffer = [];

async function processJob(job) {
  taskBuffer.push(job.data);

  if (taskBuffer.length >= 100) {
    // Perform one bulk database write instead of 100 individual ones
    await db.collection('logs').insertMany(taskBuffer);
    taskBuffer.length = 0; // Clear the buffer
  }
}

Critical Considerations

When implementing batching, you must balance efficiency with latency. If you wait too long to fill a batch, your users may experience unacceptable delays in seeing their job results. Additionally, you must ensure your system can handle partial failures; if a batch of 100 fails, you need a robust way to identify which individual tasks within that batch caused the error or need to be retried.

Key Takeaway: Aggregating granular tasks into batches minimizes database write contention and dramatically improves throughput by reducing the overhead of individual transaction management.