Hardware-Aware Async Scheduling

In this lesson, you will learn how to minimize latency by aligning your asynchronous task execution with the physical layout of your CPU’s cache and memory controllers.

The NUMA and Cache Reality

On modern multi-socket servers, we operate under a NUMA (Non-Uniform Memory Access) architecture. Accessing data in "local" memory (attached to your CPU socket) is significantly faster than fetching it from a remote socket’s memory controller. Similarly, CPU caches (L1/L2/L3) thrive on data locality. If your async runtime migrates a task across different CPU cores, you suffer "cache warming" penalties and increased interconnect traffic.

Strategic Task Pinning

Standard async runtimes (like tokio) often use work-stealing schedulers that prioritize throughput by moving tasks to idle cores. However, in latency-sensitive systems, this is counter-productive. Instead, we use affinity to bind task groups to specific NUMA nodes or cache domains.

By using the core_affinity crate in conjunction with custom runtime executors, we can ensure that tasks working on the same memory buffer stay within the same cache hierarchy:

use core_affinity;

// Create a thread pool specifically for a NUMA node
let core_ids = core_affinity::get_core_ids().unwrap();
let my_nodes_cores = &core_ids[0..4]; // Bind to the first 4 cores

std::thread::spawn(move || {
    core_affinity::set_for_current(my_nodes_cores[0]);
    // The executor here now operates with high data locality
    run_my_async_runtime();
});

Scheduling Considerations

To optimize hardware-aware scheduling, follow these two principles:

1. Locality-Preserving Queues: Rather than a global task queue that every worker thread polls, implement per-NUMA-node queues. Workers should check their local node's queue first to keep data resident in the local L3 cache.
2. Affinity-Aware Stealing: If a worker must steal work, it should prefer stealing from a core sharing the same physical L3 cache before reaching across a NUMA boundary.

By constraining where your futures execute, you reduce the expensive "cache-coherency traffic" that happens when multiple sockets fight over the same memory address, keeping your high-performance async system predictable and performant.

Key Takeaway: To maximize performance, minimize cross-NUMA task migration by pinning related asynchronous workloads to specific CPU caches and memory domains.