In this lesson, you will learn how to quantify the performance degradation caused by memory hierarchy by calculating the Effective Cycles Per Instruction (CPI).
The Memory Penalty
When a CPU executes an instruction, it expects data to be ready in the cache. However, when a "cache miss" occurs, the processor must stall while data is fetched from slower, deeper levels of the memory hierarchy (like main RAM). Because the processor cannot finish the instruction until the data arrives, this delay directly inflates your CPI.
To calculate the performance impact, we use the following model:
Effective CPI = Ideal CPI + Memory Stall Cycles per Instruction
The Memory Stall Cycles per Instruction is calculated by multiplying the frequency of misses by the time penalty for each miss:
Memory Stall Cycles per Instruction = (Memory Accesses per Instruction) × (Miss Rate) × (Miss Penalty)
Practical Example
Imagine you have a processor with an Ideal CPI of 1.0 (assuming all hits). Your code performs a load or store instruction for 30% of its operations (Memory Accesses per Instruction = 0.3). If your L1 cache has a Miss Rate of 10% and the Miss Penalty is 50 cycles, we calculate the impact as follows:
- Calculate Stall Cycles:
0.3 (Accesses/Inst) × 0.10 (Miss Rate) × 50 (Penalty) = 1.5 cycles. - Calculate Effective CPI:
1.0 (Ideal CPI) + 1.5 (Stall Cycles) = 2.5 CPI.
In this scenario, your processor performs 2.5 times slower than the "ideal" version simply because of memory latency!
// Simplified Logic
Effective_CPI = Ideal_CPI + (Accesses_Per_Inst * Miss_Rate * Miss_Penalty)
By observing this result, we see that even a small miss rate (10%) can dramatically reduce throughput when the miss penalty is high. This calculation is vital in embedded systems, where you must decide if adding a larger cache or utilizing faster, more expensive SRAM is necessary to meet your real-time performance deadlines. By quantifying this, you can justify hardware architectural choices before building your prototype.
Key Takeaway: The effective CPI is the sum of the ideal cycle count and the stall cycles incurred by cache misses, demonstrating that memory latency is often the primary bottleneck in processor performance.