In this lesson, you will learn how to identify and quantify the time the CPU spends executing hard and soft interrupt handlers to diagnose sources of system-wide jitter and latency spikes.
Understanding the Interrupt Bottleneck
When hardware needs attention, it triggers a "hard interrupt." The CPU pauses its current task to run an Interrupt Service Routine (ISR). To keep the system responsive, the kernel offloads time-consuming tasks to "soft interrupts" (softirqs), which are processed later in a deferred context. If your system experiences high interrupt latency, it means these handlers are hogging CPU cycles, preventing user-space threads from running.
We track this by observing the time spent in irq_handler_entry and irq_handler_exit (for hard IRQs) and softirq_entry and softirq_exit (for soft IRQs) using dynamic tracing.
Measuring Latency with Tracing
By leveraging the perf subsystem or BCC tools, we can measure the duration of these handlers. A high-latency spike often occurs when a specific hardware driver or network processing routine exceeds its time slice.
To gain visibility, you can hook into the kernel tracepoints. Here is a conceptual example of how we track the latency of a handler using a kprobe-style approach:
// Concept: Measuring execution time of an interrupt handler
int trace_irq_entry(struct pt_regs *ctx) {
u64 ts = bpf_ktime_get_ns();
start_times.update(&irq_id, &ts);
return 0;
}
int trace_irq_exit(struct pt_regs *ctx) {
u64 *start = start_times.lookup(&irq_id);
u64 delta = bpf_ktime_get_ns() - *start;
// Record delta in a histogram map for analysis
latency_hist.increment(bpf_log2l(delta));
return 0;
}
Interpreting the Results
When analyzing the output, look for "long-tail" latency. A single interrupt taking 500 microseconds is often negligible, but if your histogram shows a high frequency of interrupts exceeding 100 microseconds, you have identified a significant source of system instability. This usually points to drivers (e.g., NICs or storage controllers) that are performing too much work in the interrupt context rather than deferring it to a kernel thread.
Key Takeaway: High interrupt latency is often a symptom of "interrupt storms" or poorly performing ISRs that starve the scheduler; quantifying these via tracepoints is the first step toward offloading tasks or pinning IRQs to dedicated cores.