Course contentsShow
← Linux
Lesson 1608 of 1,81914. Observability and Performance TroubleshootingFree lesson

Working with kprobes and uprobes

Distinguish between kernel and user-space probes to target specific instrumentation points in software execution paths.

In this lesson, you will learn how to distinguish between kprobes and uprobes to pinpoint execution bottlenecks, whether they reside in the kernel or in your user-space applications.

When troubleshooting complex performance issues in a containerized environment, you often need to "see" exactly what a process is doing without modifying the source code or restarting the service. This is where dynamic tracing via probes comes in.

Kprobes: Probing the Kernel

kprobes are designed to intercept execution flow within the Linux kernel itself. If you suspect that a performance degradation is caused by kernel-side system calls, network stack processing, or filesystem I/O, you use a kprobe.

When you place a kprobe on a kernel function (like do_sys_open), the kernel dynamically replaces the first instruction of that function with a breakpoint instruction. When the CPU hits this, it jumps to your handler, executes your monitoring logic, and then resumes the original code path.

Uprobes: Probing User-space

Conversely, uprobes are used for instrumenting user-space applications (like your containerized services or databases). If you want to know how many times a specific function in your C++ or Go application is called, or what values are being passed to a specific library, you attach a uprobe to the application's binary on disk.

The kernel handles this by creating a breakpoint in the process's address space. When your application execution hits that address, the kernel traps the event. This is incredibly powerful for debugging closed-source binaries or tracking function latency in production.

Practical Distinction

Think of it as a choice of "where" the problem lives:

  • Use a kprobe if you need to know why the kernel is taking too long to handle a socket buffer or why a process is stuck in a D (uninterruptible sleep) state.
  • Use a uprobe if you need to profile your business logic, such as counting how many times an OrderProcess() function is triggered within your application container.

Key Takeaway: Use kprobes to hook into kernel-space execution paths for system-level observability, and use uprobes to instrument user-space application logic without requiring binary recompilation.