Streaming Data Deduplication

Streaming Data Deduplication: The Rolling Window Filter

In this lesson, you will learn how to identify and prune redundant data packets as they flow through a pipeline using a memory-efficient rolling window filter.

The Problem of Stream Redundancy

When consuming high-velocity data streams, you often encounter duplicate chunks due to network retries or producer-side logic errors. If you store or process these twice, you waste compute and storage resources. To solve this, we use a Rolling Window Filter—a stateful mechanism that tracks a limited "memory" of recently seen data hashes.

Implementing the Rolling Window

Instead of checking every chunk against a massive database (which would be too slow), we maintain a small Set (or a Map with timestamps) that acts as a sliding window. As each chunk arrives, we generate a hash or unique identifier. If the ID exists in our window, we drop it; if not, we pass it downstream and add the ID to our set.

Here is how you can implement this inside a Transform stream:

const { Transform } = require('stream');
const crypto = require('crypto');

const deduplicator = new Transform({
  objectMode: true,
  transform(chunk, encoding, callback) {
    const hash = crypto.createHash('md5').update(JSON.stringify(chunk)).digest('hex');
    
    // Check if we've seen this in our window
    if (this.seen.has(hash)) {
      return callback(); // Drop duplicate: don't push anything
    }

    // Add to window
    this.seen.add(hash);
    this.push(chunk);
    
    // Prevent memory leaks: prune window size
    if (this.seen.size > 1000) {
      const firstKey = this.seen.keys().next().value;
      this.seen.delete(firstKey);
    }
    
    callback();
  }
});

// Initialize the state
deduplicator.seen = new Set();

Why a Rolling Window?

The Set size limit is crucial. By keeping only the last 1,000 unique items, we ensure that our memory footprint remains constant regardless of how long the stream runs. This respects your system's memory limits while providing high-performance filtering. If a duplicate arrives outside of this "window," it is treated as a new piece of data, which is usually the desired behavior for long-lived streams where exact global state is unnecessary.

Key Takeaway: Use a fixed-size Set within a Transform stream to track recent hashes, effectively dropping duplicates while maintaining a predictable, memory-safe footprint.