In this lesson, you will learn how to identify when kernel threads jump across NUMA nodes and how this cross-socket movement degrades memory access performance.
In a NUMA (Non-Uniform Memory Access) architecture, a processor accesses its local memory much faster than memory connected to a remote socket. When the Linux scheduler migrates a thread from CPU A (on Node 0) to CPU B (on Node 1), the thread’s "hot" cache lines—previously populated in Node 0’s local memory—become distant, causing a spike in latency and cache misses.
Identifying Migration Events
To detect this, we tap into the sched:sched_migrate_task tracepoint. This event fires whenever the scheduler moves a task from one runqueue to another. By tracking the dest_cpu and mapping it to its associated NUMA node, we can flag migrations that cross node boundaries.
Measuring Cache Impact
Once you detect a cross-node migration, you must correlate it with hardware events. We typically use perf or a BPF-based approach to monitor LLC-load-misses. If a high rate of LLC (Last Level Cache) misses occurs immediately following a sched_migrate_task event, you have confirmed that the thread is suffering from "remote memory access penalty."
Consider this logic in an eBPF program:
SEC("tracepoint/sched/sched_migrate_task")
int trace_migration(struct trace_event_raw_sched_migrate_task *ctx) {
int src_node = cpu_to_node(ctx->orig_cpu);
int dst_node = cpu_to_node(ctx->dest_cpu);
if (src_node != dst_node) {
// Increment BPF map to track migration count per-process
// Trigger alert or log stack trace to analyze memory locality
bpf_printk("Migration detected: Task %d moved nodes %d -> %d\n",
ctx->pid, src_node, dst_node);
}
return 0;
}
By aggregating these events, you can pinpoint specific kernel threads that are "bouncing" across sockets—a common bottleneck in high-throughput database or networking applications. If you observe high migration frequency, it is often a signal to apply CPU affinity (taskset) or refine your memory allocation policy (numactl) to keep data and execution units closer together.
Key Takeaway: Cross-NUMA migration forces threads to abandon local caches for remote memory, causing a measurable latency penalty that should be diagnosed by correlating scheduler events with cache-miss metrics.