In this lesson, you will learn how to identify and quantify cross-socket memory access patterns that degrade performance in multi-socket Linux systems.
Understanding NUMA Locality
Non-Uniform Memory Access (NUMA) architectures divide system memory into "nodes" associated with specific CPU sockets. While a CPU can access memory from a remote node (a "remote hop"), doing so incurs significant latency penalties compared to accessing local memory. When your application threads frequently fetch data stored in a remote NUMA node, the increased interconnect traffic creates a performance bottleneck that often manifests as high average memory latency.
Detecting Remote Access
To analyze this, we focus on the hardware-level monitoring provided by Performance Monitoring Units (PMU). We specifically look for the difference between local and remote memory hits. Using perf, you can track node-loads and node-load-misses.
A high ratio of node-load-misses relative to total loads is a strong indicator that your processes are not "NUMA-aware." If you see high remote access, your application is likely suffering from interconnect contention—the data bus between sockets is saturated, leading to stalled execution cycles.
Monitoring with Perf
You can observe these patterns across your sockets using the following command to sample memory traffic:
perf stat -a -e node-loads,node-load-misses sleep 5
If node-load-misses is a significant fraction of node-loads, it suggests that data is allocated on a node distant from the CPU executing the thread. In complex multi-threaded applications, this is often caused by threads being migrated across sockets by the scheduler after the memory has already been pinned to a specific node, or by improper thread-to-memory affinity settings.
To mitigate this, developers often use numactl --membind or numactl --cpunodebind to force process affinity, or implement "NUMA-aware" memory allocation patterns where threads allocate memory from the same node they reside on. By aligning your thread execution with the physical memory it consumes, you minimize the latency penalties inherent in multi-socket traversal.
Key Takeaway: High remote NUMA hits signal that your threads are crossing socket boundaries to access data, introducing interconnect latency that can be mitigated by enforcing stricter CPU and memory affinity.