In this lesson, you will learn how hardware multithreading prevents the CPU from sitting idle during long-latency events by interleaving multiple instruction streams.
The Problem: The Latency Gap
As we’ve discussed in our memory hierarchy lessons, a cache miss is catastrophic. If a load instruction triggers a DRAM access, the pipeline might stall for hundreds of cycles. In a standard superscalar processor, this "stall" causes the entire core to grind to a halt. Hardware multithreading solves this by maintaining multiple sets of architectural state (registers, program counters) on a single physical core, allowing the CPU to switch to a different thread the moment one hits a wall.
Fine-Grained Multithreading (FGMT)
Fine-grained multithreading uses a "round-robin" approach. The CPU switches between threads on every single clock cycle.
- How it works: If Thread A is waiting for a memory fetch, the processor simply skips it in the rotation and executes an instruction from Thread B or C.
- The Benefit: It effectively hides short stalls. Because the threads are interleaved, the probability that all threads are waiting on a cache miss at the same time is extremely low.
Simultaneous Multithreading (SMT)
SMT takes this a step further by leveraging our superscalar execution units. Instead of switching threads on a cycle-by-cycle basis, SMT allows the CPU to issue instructions from multiple threads simultaneously in the same clock cycle.
Imagine a pipeline width of 4 instructions:
Cycle 1: [Instr A1] [Instr A2] [Instr B1] [Instr C1]
In this example, instructions from Thread A, B, and C occupy the execution ports at the exact same time. SMT treats all threads as a single pool of instructions, filling gaps in the execution units that would otherwise be wasted by the limited Instruction Level Parallelism (ILP) of a single thread.
Comparison
- FGMT simplifies hardware but forces a rigid switching pattern.
- SMT is complex but maximizes resource utilization by dynamically grabbing ready instructions from any thread to keep all functional units busy.
By keeping multiple "contexts" ready in the hardware, we ensure that when one thread stalls, the core remains productive, effectively "hiding" the latency of memory hierarchy bottlenecks.
Key Takeaway: Hardware multithreading masks memory latency by maintaining multiple thread contexts, ensuring the CPU core stays busy even when one stream is blocked by a cache miss.