In this lesson, you will learn how to pinpoint the exact kernel function responsible for silently dropping network packets using dynamic tracing tools.
When a packet fails to reach your application, it often disappears deep within the kernel stack—long after the NIC has received it, but before it reaches a socket. Relying on netstat -s or nstat tells you that drops are occurring, but not why. To find the culprit, we must observe the kernel's execution flow in real-time.
Using trace-cmd
trace-cmd is a powerful front-end for the Linux ftrace subsystem. When you suspect packets are being dropped, you can trace the kfree_skb function, which is the standard kernel routine for releasing a socket buffer (skb). Whenever the kernel decides a packet is invalid, malformed, or rejected by a policy, it calls kfree_skb.
To start tracking these events, run:
# Record all stack-level drops for 10 seconds
sudo trace-cmd record -e skb:kfree_skb sleep 10
# View the results to see which function triggered the drop
sudo trace-cmd report
The output will show you the "location" (the kernel instruction pointer) of the drop. You can then map this address to a function name using vmlinux or nm to identify if the drop happened due to a firewall rule, an invalid TCP flag, or a routing table mismatch.
Using systemtap
If you need deeper context—such as the specific packet headers or the interface name at the moment of the drop—systemtap allows you to write scripts that capture kernel state. By hooking into kfree_skb, you can extract the drop location:
probe kernel.function("kfree_skb") {
printf("Packet dropped at: %s\n", symdata($location))
}
By correlating these trace points with your specific network conditions, you can distinguish between a hardware-level discard and an intentional kernel policy drop. This removes the guesswork from debugging complex network failures where traffic seems to vanish into thin air.
Key Takeaway: Use trace-cmd or systemtap to hook into kfree_skb to capture the kernel's stack trace at the exact moment a packet is discarded, revealing which internal function triggered the drop.