In this lesson, you will learn how CPU caches use patterns in the memory addresses accessed by your code to anticipate future data needs and hide memory latency.
To a CPU, main memory (DRAM) is slow, while the cache is lightning-fast. Since we want to keep the processor moving at full speed, the hardware relies on two fundamental behaviors found in almost all software: Temporal Locality and Spatial Locality.
Temporal Locality: The "Near Future" Habit
Temporal locality is the principle that if you access a specific memory address, you are likely to access that same address again very soon. Think of a loop index or a frequently used variable.
int sum = 0;
for (int i = 0; i < 100; i++) {
sum += data[i]; // 'sum' is accessed 100 times
}
The hardware exploits this by keeping sum in the cache after the first access. Instead of reaching out to the slow DRAM every single iteration, the CPU simply grabs sum from the nearby, fast cache.
Spatial Locality: The "Neighbor" Habit
Spatial locality is the principle that if you access a memory address, you are likely to access its neighbors shortly after. This is most obvious when iterating through an array. When you load data[0], the hardware doesn't just fetch that one integer; it pulls a whole "cache line" (a block of adjacent memory) into the cache.
for (int i = 0; i < 100; i++) {
process(data[i]);
}
Because the hardware brought data[1], data[2], and data[3] into the cache along with data[0], the CPU finds them waiting when it needs them. The hardware essentially "predicts" that your next step will be the adjacent memory address, successfully hiding the latency that would have occurred if it had to fetch every single index from main memory individually.
By design, caches turn "random" memory requests into predictable, high-speed streams, ensuring that your pipeline stages—which we've worked so hard to optimize—aren't constantly waiting for data to crawl in from main memory.
Key Takeaway: Caches leverage temporal locality to keep frequently used data nearby and spatial locality to pre-fetch neighboring data before the CPU even asks for it.