In this lesson, you will learn how to identify when the Linux scheduler migrates a thread from one physical CPU core to another, a behavior that often destroys performance for cache-sensitive applications.
The Problem with Scheduler Migration
In an ideal high-performance environment, a thread stays pinned to a specific core. When a thread moves across cores, it loses its "warm" L1 and L2 caches, forcing the processor to fetch data from slower levels of the memory hierarchy. While the scheduler's default balancing logic is generally beneficial for generic workloads, it can be detrimental to latency-critical tasks like high-frequency trading or real-time signal processing.
Tracing the Shift with eBPF
To observe these shifts, we need to trace the kernel's scheduler events. Specifically, we monitor the sched_migrate_task tracepoint. This event fires whenever the kernel decides a thread should be moved to a different runqueue (and thus, a different CPU).
You can use bpftrace to capture these events in real-time. The following one-liner logs the PID, the command name, the source CPU, and the destination CPU:
sudo bpftrace -e 'tracepoint:sched:sched_migrate_task {
printf("Thread %d (%s) moved from CPU %d to %d\n",
args->pid, args->comm, args->orig_cpu, args->dest_cpu);
}'
Interpreting the Results
When you run this script while your application is under load, you might see a flurry of migrations. If you see a specific thread ID jumping between different cores repeatedly, it is a sign that your application is suffering from "cache thrashing."
This behavior often occurs because the kernel detects a CPU load imbalance and attempts to distribute the work. However, if your application requires thread-local data to reside in cache, these migrations will manifest as sudden spikes in instruction cycles per instruction (CPI) or increased memory latency. Once identified, you can mitigate these shifts using the sched_setaffinity system call to restrict your process threads to specific, dedicated cores, ensuring they never migrate away from their data.
Key Takeaway: Thread affinity shifts degrade cache locality and increase latency; by using eBPF to monitor the sched_migrate_task tracepoint, you can confirm if the scheduler is disrupting your performance and justify pinning threads to specific cores.