Filesystem Event Monitoring with Inotify
In this lesson, you will learn how to leverage the inotify subsystem to track file system modifications across complex, layered container images.
When managing containers, understanding how processes interact with the underlying root filesystem—especially when utilizing OverlayFS—is critical. inotify allows you to register watches on specific inodes or directories, providing an asynchronous notification stream via a file descriptor whenever events like IN_MODIFY, IN_CREATE, or IN_DELETE occur.
Monitoring Across Layers
Container runtimes often use OverlayFS to stack multiple read-only image layers beneath a writable "upper" layer. A common challenge is determining which layer is being modified by a containerized application. While inotify tracks the inode as it appears in the merged mount point, you can enhance your observability by pairing it with procfs metadata to correlate these events.
Consider a monitoring agent running in the host namespace that attaches to the directory where the container's upper layer resides. Using the inotify_add_watch system call, you can identify precisely when a container writes a configuration file:
int fd = inotify_init();
int wd = inotify_add_watch(fd, "/var/lib/docker/overlay2/<id>/diff", IN_MODIFY | IN_CREATE);
while (1) {
char buffer[4096];
read(fd, buffer, sizeof(buffer));
// Process event structures to determine which file changed
}
Constraints and Performance
Because inotify is bound to the kernel’s event queue, it is highly efficient, but it does have limitations when dealing with massive, deep-directory structures common in modern microservices. If you attempt to watch an entire recursive directory tree, you must manually iterate through subdirectories, as inotify does not support recursive watches natively.
For high-throughput environments, be mindful of the fs.inotify.max_user_watches kernel parameter; container orchestration platforms often trigger an excessive number of watches, leading to "no space left on device" errors if the system limit is too low. When combined with your previous knowledge of perf events or eBPF, inotify provides the specific file-level context that system-wide tracing often lacks.
Key Takeaway: Inotify provides a lightweight mechanism to hook into filesystem mutations, enabling granular tracking of file modifications within container layers when properly scoped to prevent kernel limit exhaustion.