In this lesson, you will learn how to visualize system call frequency distributions to identify anomalous application behavior by shifting from raw event counting to statistical profiling.
Beyond Mean Latency: The Power of Distribution
While average system call latency or aggregate count metrics are useful, they often hide the "shape" of application behavior. An application might appear performant on average, but a bimodal distribution of read() or write() calls could indicate that the process is alternating between fast memory cache hits and slow disk I/O, or worse, thrashing due to unexpected synchronization bottlenecks.
To analyze these distributions at scale, we collect per-process system call counts and map them into BPF histograms. Instead of streaming every event to user-space—which incurs massive overhead—we perform the aggregation in the kernel using BPF_MAP_TYPE_HASH or BPF_MAP_TYPE_PERCPU_HASH.
Visualizing Anomalies
When you plot these distributions (usually using log-linear or power-of-two buckets), you are looking for deviations from the "baseline" signature. For example, a web server usually exhibits a very consistent histogram for epoll_wait and read. If you suddenly observe a "long tail" or a secondary spike in the distribution, it suggests the application has entered an unintended state—perhaps due to a configuration drift, a file descriptor leak, or a sudden change in client request patterns.
Example: Tracking syscall counts
Using a simple BPF map, you can track the frequency of syscalls per process:
// BPF map to track syscall frequencies
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__uint(max_entries, 10240);
__type(key, u32); // syscall ID
__type(value, u64); // counter
} syscall_freq_map SEC(".maps");
SEC("tracepoint/raw_syscalls/sys_enter")
int trace_sys_enter(struct trace_event_raw_sys_enter *ctx) {
u32 syscall_id = ctx->id;
u64 *val = bpf_map_lookup_elem(&syscall_freq_map, &syscall_id);
if (val) {
__sync_fetch_and_add(val, 1);
}
return 0;
}
By snapshotting this map at regular intervals and calculating the delta between observations, you can detect when an application’s system call profile changes, allowing you to catch regressions that raw throughput metrics would miss.
Key Takeaway: Monitoring the shape of system call distributions allows you to identify performance regressions and anomalous state transitions that remain hidden within aggregate performance metrics.