BiteSizedChunks.comLearn one small thing at a time.
Course contentsShow
← Linux
Lesson 511 of 1,8195. Process Management and SignalingFree lesson

Scheduler Latency Statistics

Use performance monitoring tools to identify processes causing system-wide latency spikes.

In this lesson, you will learn how to identify the specific processes causing system-wide latency spikes by analyzing scheduler statistics.

Understanding Scheduler Latency

In a multitasking system, "scheduler latency" occurs when a process is ready to run but must wait for the CPU to become available. If a process stays in a "runnable" state for too long before actually executing, we call this a scheduling delay. High latency leads to perceived system sluggishness, jitter in multimedia, or slow response times in network services.

Monitoring with perf

To pinpoint the culprits, we use the perf tool, which hooks into the kernel's tracepoints to record how long tasks spend waiting in the run-queue. By examining these statistics, we can see if a specific process is hogging the CPU or if many processes are competing, forcing others to wait.

You can capture these statistics using the following command:

# Record scheduler latency for 5 seconds
sudo perf sched record sleep 5

# Analyze the recorded data to see which tasks are delayed
sudo perf sched latency

The perf sched latency output provides a clear table showing:

  • Task: The process name and PID.
  • Runtime: The total time the process spent on the CPU.
  • Switches: How many times the process was context-switched.
  • Avg delay: The average time the process spent waiting in the queue before getting CPU time.

Identifying the Bottleneck

When you run this analysis, look specifically at the Avg delay column. If a process shows a very high average delay, it indicates that it is being frequently preempted or waiting behind other heavy processes. This is often caused by a "runaway" process with high priority or an excessive number of threads competing for the same cores. By identifying the top offenders, you can decide whether to adjust their priority, move them to different CPU cores using affinity, or throttle them using cgroups.

Key Takeaway: Scheduler latency statistics help you bridge the gap between "the system feels slow" and "this specific process is waiting too long for the CPU."