In this lesson, you will learn how to enforce memory affinity on multi-socket servers to minimize latency in high-performance computing (HPC) workloads.
On modern multi-socket servers, we utilize Non-Uniform Memory Access (NUMA). Each CPU socket has its own local memory banks. While a CPU can access memory attached to a different socket, it incurs a significant "remote access" latency penalty. To achieve peak performance, your processes must prioritize "local" memory allocation.
The Default Policy: First-Touch
By default, the Linux kernel uses the "first-touch" policy. The kernel allocates a page of memory on the node where the thread first writes to that memory. If your initialization code runs on CPU node 0, that thread’s memory is pinned to node 0. If that thread then migrates to CPU node 1, it will experience high latency accessing its data across the interconnect (e.g., QPI or UPI).
Enforcing Affinity with numactl
For HPC tasks, we use the numactl utility to explicitly define how memory should be allocated. You can restrict a process to run on specific nodes and force its memory to be allocated locally:
# Run a process, pinning it to CPU node 0 and forcing local memory allocation
numactl --cpunodebind=0 --membind=0 ./hpc_application
If your application requires a strictly local approach, the --localalloc flag is preferred, as it instructs the kernel to prefer the node where the task is currently running, preventing it from spilling over into distant memory banks if the local node has capacity.
Monitoring Contention
When tuning these policies, remember that improper pinning can lead to resource starvation. If you bind a high-memory application to a single node, you bypass the kernel's ability to balance memory across the system, which can trigger the OOM Killer prematurely if that specific node hits its capacity, even if other nodes have gigabytes of free RAM. Always monitor /proc/buddyinfo and numastat to ensure your workload isn't hitting "remote" allocation counts, which indicate that your memory policy is failing to keep data local.
Key Takeaway: To maximize performance in multi-socket systems, use numactl to align memory allocation with CPU scheduling, ensuring data remains physically close to the processor executing the task.