Compacting Message Logs

In this lesson, you’ll learn how to optimize your message logs by collapsing redundant state changes to ensure your system recovers quickly and efficiently.

When handling high-frequency streams—such as tracking a user's balance or an inventory count—your message log can grow indefinitely. If a worker crashes and needs to rebuild its state by replaying every single "Set Balance" message from the start of time, startup latency becomes unacceptable. This is where Log Compaction comes in.

The Concept

Instead of storing every incremental update, a compacted log keeps only the latest value for a specific key (e.g., user_123). If you have a log with ten updates to the same user's account, you don't need the first nine updates to determine the current state; you only need the final one.

In a log-based messaging system, the "Compactor" process periodically scans the log, identifies keys with multiple entries, and discards all but the most recent version.

Implementing Log Compaction Logic

Imagine a simple log of state changes represented as objects. We can simulate compaction by reducing an array of messages to the latest value per key:

const messageLog = [
  { key: 'user_1', value: 100 },
  { key: 'user_2', value: 50 },
  { key: 'user_1', value: 150 }, // Latest for user_1
  { key: 'user_2', value: 75 }   // Latest for user_2
];

function compactLog(log) {
  const latestStates = new Map();
  // We iterate through the log; later entries overwrite earlier ones
  for (const entry of log) {
    latestStates.set(entry.key, entry.value);
  }
  return Array.from(latestStates, ([key, value]) => ({ key, value }));
}

console.log(compactLog(messageLog));
// Output: [{ key: 'user_1', value: 150 }, { key: 'user_2', value: 75 }]

Why This Matters for Workers

By implementing compaction, your background workers don't need to process the entire history of a specific entity. They can process a "snapshot" of the compacted log to reach the current state instantly, then continue consuming only the new, incoming delta changes. This significantly reduces memory usage and speeds up recovery times in your distributed task architecture.

Key Takeaway: Log compaction prevents infinite growth and minimizes system recovery time by keeping only the most recent state for each unique key in your message stream.