Understanding CPU Cache Hierarchies

Map software data access patterns to L1, L2, and L3 cache constraints.

In this lesson, you will learn how to map your Rust data access patterns to the physical constraints of L1, L2, and L3 cache hierarchies to minimize latency.

Modern CPUs hide the massive latency gap between the CPU and main memory (RAM) using a multi-layered cache hierarchy. Understanding where your data "lives" during execution is the difference between a high-performance system and one bottlenecked by memory stalls.

The Cache Hierarchy Landscape

L1 Cache (Private/Core): Extremely fast (~1-4 cycles), typically split into instruction and data caches. This is where your "hot" variables and tight loops must reside.
L2 Cache (Private/Core): Larger than L1 (~10-15 cycles), holding frequently accessed data that exceeds L1 capacity.
L3 Cache (Shared/Package): Much larger (~40-70 cycles), shared across all cores. Accessing data here is significantly costlier than L1/L2.

Mapping Data to Latency

When you design data structures, you are essentially determining the "cache pressure." If your hot loop traverses a large structure, you risk cache misses, forcing the CPU to fetch data from L3 or even main memory (which can take 200+ cycles).

Consider a scenario where you are processing a large vector of structs:

struct Particle {
    x: f32, y: f32, z: f32, // 12 bytes
    velocity: f32,          // 4 bytes
    id: u64,                // 8 bytes (Padding/Alignment adds more)
}

If your algorithm only needs x, y, and z, iterating over a Vec<Particle> forces the CPU to pull the id and velocity into the L1 cache needlessly. This reduces the effective capacity of your L1 cache, causing earlier evictions of useful data. By shifting to a Structure-of-Arrays (SoA) layout—where x, y, and z are in separate contiguous arrays—you ensure that your L1 cache lines are packed exclusively with the data relevant to your current calculation.

This spatial locality allows the CPU prefetcher to load subsequent cache lines before you even ask for them, keeping the pipeline fed and avoiding the "memory wall."

Key Takeaway: Optimize your data structures so that frequently accessed fields are packed contiguously, ensuring that your most critical working sets fit comfortably within the L1/L2 cache boundaries.