In this lesson, you will learn how to use ftrace to identify the precise sources of latency within your kernel’s interrupt handlers and scheduling sequences.
Understanding Latency with Ftrace
When building embedded systems, especially those with real-time requirements, "jitter"—unpredictable delays in task execution—is the enemy. ftrace is the Linux kernel’s built-in tracer, designed to track the flow of function calls and state transitions with minimal overhead. Unlike perf, which samples the system, ftrace logs events as they occur, making it perfect for capturing the exact moment an interrupt is fired or a process is preempted.
Measuring Latency
To analyze latency, we primarily use the function_graph tracer. This provides a visual representation of how long functions take to execute. To start tracing a specific system state (e.g., how long a task spends waiting for a CPU), you interact with the tracefs filesystem, typically mounted at /sys/kernel/tracing.
For example, to monitor the latency of an interrupt handler or a specific kernel event, you can configure the tracer like this:
# Prepare the tracer
cd /sys/kernel/tracing
echo function_graph > current_tracer
# Trace only relevant events (e.g., scheduler)
echo sched_switch > set_ftrace_filter
# Start tracing and view the output
echo 1 > tracing_on
cat trace | head -n 50
The output displays the execution time of nested functions in a C-like format. You will see a + or ! marker next to function calls that exceed specific time thresholds. A ! indicates a significant latency spike, which is your primary indicator that a kernel thread or hardware interrupt is blocking the CPU longer than desired.
By filtering for specific kernel subsystems—such as irq_handler_entry or sched_wakeup—you can pinpoint whether a latency issue is caused by a long-running driver, a hardware interrupt that isn't being acknowledged fast enough, or priority inversion between processes. This granular visibility allows you to distinguish between "hard" latency (hardware/IRQ related) and "soft" latency (scheduler/task priority related).
Key Takeaway: Use ftrace’s function_graph tracer to capture real-time execution flow and identify specific functions causing unacceptable delays in your kernel’s critical paths.