In this lesson, you will learn how to overcome the "Memory Wall"—the performance bottleneck caused by the massive speed gap between fast CPUs and slow DRAM—by using non-blocking caches and memory interleaving.
Even with multi-level cache hierarchies, a cache miss forces the CPU to wait for data from the main memory. Because DRAM access latency is significantly higher than a single CPU clock cycle, the processor often sits idle, effectively hitting a "wall." To keep the pipeline moving, we must allow the system to continue processing other instructions while memory requests are pending.
Non-blocking Caches
A standard cache "blocks" the entire pipeline during a miss until the data returns. A non-blocking cache (or "hit-under-miss" cache) allows the CPU to continue executing instructions that don't depend on the missing data. It uses something called a Miss Status Holding Register (MSHR) to keep track of outstanding requests. As long as the CPU finds a "hit" in the cache while the previous "miss" is still being fetched from DRAM, the execution continues.
Memory Interleaving
Another bottleneck occurs because DRAM can only handle one access at a time. Memory Interleaving splits the physical memory into multiple independent banks. By mapping sequential addresses across different banks, the system can initiate a new access request to Bank B while Bank A is still busy finishing a previous request.
Consider this sequence:
// Array access pattern
for (int i = 0; i < 1000; i++) {
sum += data[i];
}
Without interleaving, the CPU waits for the entire latency of a single DRAM bank for every request. With interleaving, the hardware can overlap the "busy" cycles of Bank 0, Bank 1, and Bank 2. By the time the CPU needs data[2], the memory controller has already begun the "precharge" and "activate" phases for that bank in the background.
By decoupling the CPU execution flow from the slow serial nature of DRAM, these techniques ensure the processor remains productive even when the data it needs isn't immediately available in the local cache.
Key Takeaway: You can minimize the Memory Wall by using non-blocking caches to permit execution during misses and memory interleaving to overlap latency across multiple memory banks.