Streaming Data Pipelines

In this lesson, you will learn how to architect high-throughput streaming pipelines that maintain peak performance by sizing data chunks to fit comfortably within your CPU’s L3 cache.

The L3 Cache Bottleneck

In streaming data pipelines, processing speed is often dictated by memory bandwidth rather than compute power. When you process individual items one by one, you frequently incur the cost of fetching data from main RAM, which is orders of magnitude slower than the L3 cache. Even worse, if your working set exceeds the L3 cache size, your CPU stalls while waiting for memory controllers, leading to a "memory wall."

To avoid this, we transition from individual item processing to chunked batching. By grouping data into blocks that fit within the L3 cache (typically between 1MB and 8MB, depending on your architecture), you ensure that once a chunk is pulled into the cache, every operation—filtering, transformation, or aggregation—happens at silicon speed without further trips to RAM.

Architectural Implementation

Instead of a simple iterator processing a stream of T, we process Vec<T> chunks.

const CHUNK_SIZE: usize = 1024; // Tuned for L3 cache density

fn process_pipeline(input_stream: impl Iterator<Item = Data>) {
    for chunk in input_stream.chunks(CHUNK_SIZE) {
        // 'chunk' is now local to L3. 
        // We perform multiple passes (map, filter, aggregate) 
        // while the data is still 'hot' in the cache.
        let results = chunk
            .map(transform)
            .filter(predicate)
            .collect::<Vec<_>>();
        
        process_results(results);
    }
}

Designing for Locality

When designing these pipelines, prioritize Data-Oriented Design. If your chunks contain large structs, the L3 cache will fill up faster than if you use a "Structure of Arrays" (SoA) layout. By aligning your chunk size with the cache line and keeping the working set size below the L3 threshold, you minimize cache evictions.

Remember: the goal is to keep the pipeline "fed" with chunks that reside entirely within the cache, allowing the CPU to execute instructions on data that is already waiting in the L1/L2/L3 hierarchy.

Key Takeaway: Architect your streaming pipeline to process data in chunks sized specifically to fit within your CPU’s L3 cache to minimize expensive round-trips to main memory.