In this lesson, you will learn how to detect when your application threads are stalling because they are stuck waiting for mutexes held by other threads.
The Problem of Mutex Contention
A mutex (mutual exclusion) is a synchronization primitive used to ensure that only one thread accesses a shared resource, like a memory buffer or a data structure, at a time. While necessary for data integrity, mutexes can become a bottleneck. When a thread attempts to acquire a mutex that is already held by another thread, the kernel forces the waiting thread to "sleep" (or block) until the lock is released.
If your application has high contention, you will see many threads in a D (uninterruptible sleep) or S (interruptible sleep) state, even if your CPU usage is not at 100%. This is because the threads aren't performing work—they are simply waiting in a queue.
Detecting Contention with perf
To identify if your program is spending significant time waiting for locks, we use the perf tool, which hooks into the kernel’s lock-tracking infrastructure. You can record the lock statistics of a running process with:
# Enable lock statistics and record for 10 seconds
sudo perf record -e 'lock:contention_begin' -a sleep 10
After the recording, you can view the report to see which functions are causing the most wait time:
sudo perf report
Look for functions related to mutex_lock or internal kernel synchronization primitives. If you see high "wait time" values associated with a specific shared resource in your code, it indicates that your threads are frequently fighting for access.
Why this happens
Contention usually occurs when:
- Critical sections are too large: The code inside the lock takes too long to execute.
- Lock granularity is too coarse: You are using one giant mutex for an entire data structure instead of smaller, per-item locks.
- High thread count: Too many threads are trying to access the same limited resource simultaneously.
By identifying these "wait hot spots," you can refactor your code to release locks sooner or distribute the workload across more independent mutexes, allowing your threads to run in parallel rather than standing in line.
Key Takeaway: Mutex contention happens when threads block while waiting for shared access; you can identify these bottlenecks by using perf to measure kernel-level lock wait times.