In this lesson, you will learn how to identify the specific locks or resources causing threads to hang in an uninterruptible sleep (D state) during resource starvation.
The Anatomy of the D State
When a process enters an uninterruptible sleep state, it is typically waiting for a synchronous I/O operation or a kernel-level lock to release. Unlike processes waiting for CPU time, these threads are "stuck" because they are blocked on a dependency that has failed to materialize. When this happens across multiple threads, the system suffers from starvation: the kernel cannot move forward because the resources required for completion are held indefinitely by a stalled holder.
Locating the Bottleneck
To diagnose this, we must link the blocked thread to the specific resource it is awaiting. We look for threads in TASK_UNINTERRUPTIBLE and examine their kernel stack trace to identify the mutex, semaphore, or spin_lock call site.
You can inspect the wait chain using task analysis in drgn or by checking the /proc/<pid>/stack interface. For a running system, a simple way to identify the culprit is to look for threads blocked on the same symbol:
# Identify the kernel functions where threads are stuck
for pid in $(ps -eo pid,state | awk '$2=="D"{print $1}'); do
echo "Process $pid stack:"
cat /proc/$pid/stack
done
If multiple processes are stuck in xfs_file_buffered_write or similar filesystem paths, they are likely competing for a common inode lock or buffer head. The key is to find the "holder"—the thread that currently possesses the lock but is failing to release it, possibly due to a hardware hang or an extremely long-latency I/O operation (like a stalled disk controller).
Once the offending lock is identified via the stack trace, you can use bpftrace to monitor the duration of the lock acquisition attempt:
// BPF snippet to trace lock wait times
kprobe:mutex_lock { @start[tid] = nsecs; }
kretprobe:mutex_lock /@start[tid]/ {
@duration = hist(nsecs - @start[tid]);
delete(@start[tid]);
}
By correlating the high-latency lock acquisitions with the D state processes, you can pinpoint exactly which kernel resource is the source of the starvation.
Key Takeaway: Threads in uninterruptible sleep are not waiting for CPU; they are blocked by kernel-level dependencies, and identifying them requires tracing the specific lock acquisition points in their stack traces to find the stalled resource holder.