Page fault debugging

In this lesson, you will learn how to identify and remediate excessive page fault scenarios using kernel tracing tools to pinpoint memory-intensive bottlenecks.

Understanding Page Faults

A page fault occurs when a process attempts to access a page of memory that is not currently mapped into its physical address space. While minor page faults are a standard part of demand paging, major page faults—which require the kernel to fetch data from disk—can lead to severe I/O wait times and latency spikes. When an application experiences excessive faulting, it often suggests thrashing or inefficient memory access patterns.

Diagnosing with perf

To debug these scenarios, we use perf, the standard Linux performance counter tool. We monitor the page-faults event to see which processes are triggering the highest volume of faults:

# Monitor page faults for all processes system-wide
sudo perf stat -e page-faults -a sleep 5

# Trace specific process faults and map them to code locations
sudo perf record -e page-faults -p <PID> -g
sudo perf report

When you examine the output, look for high counts of major-faults. If these are high, the system is likely pulling data from swap or file-backed storage repeatedly.

Fixing Excessive Faulting

Once you identify the offending process, you have three primary tactical levers to reduce faulting:

1. Memory Locking: If the application is sensitive to latency, you can prevent specific memory regions from being swapped out using mlock(2).
2. Hugepages: If the page table overhead is the culprit (common in databases), switching to Hugepages reduces the number of page table entries the CPU must manage, effectively reducing the frequency of TLB misses and associated page faults.
3. Tuning Swap: If major faults are rampant due to low physical RAM, check your /proc/sys/vm/swappiness value. Lowering this value makes the kernel less aggressive about moving process memory to the swap partition, keeping active data in the page cache longer.

By correlating these trace points with your memory layout knowledge, you can isolate whether the application needs more physical memory, better access locality, or a shift in the kernel’s reclamation strategy.

Key Takeaway: Use perf to distinguish between routine minor page faults and performance-killing major faults, then mitigate the latter by optimizing memory pinning or adjusting the system's paging aggressiveness.