In this lesson, you will learn how to bypass standard cluster logs and perform raw packet inspection within a container's network namespace to resolve elusive inter-pod communication failures.
When application metrics and distributed tracing logs fail to explain why two pods cannot talk, you are often dealing with silent packet loss, unexpected TCP resets, or protocol-level mismatches. Because Kubernetes pods frequently run stripped-down images lacking diagnostic tools, you cannot simply apt install tcpdump on the fly.
The Ephemeral Debugging Container
The most reliable way to inspect traffic is to leverage the kubectl debug command. This allows you to attach a temporary "sidecar" container to an existing, running pod. This sidecar shares the network namespace of the target pod, effectively putting it "inside" the same network interface as your application.
To capture traffic on a specific pod, execute:
kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name>
Once inside the shell of the netshoot container, you can run tcpdump to observe the traffic exactly as the application sees it. For instance, to monitor traffic on port 8080:
tcpdump -i eth0 port 8080 -vv
Why This Matters
By capturing packets directly within the namespace, you strip away the abstractions of the CNI (Container Network Interface) and the kube-proxy. If you see the packet arriving at the interface via tcpdump but the application process never receives it, you have identified that the issue exists within the container's internal networking stack or application binding—not the cluster overlay. Conversely, if you see the TCP handshake start but never complete, you are likely looking at a firewall (NetworkPolicy) or a connection-tracking issue between nodes.
This "in-namespace" approach is the ultimate source of truth because it eliminates variables introduced by host-level monitoring tools, which often struggle to map encapsulated traffic (like VXLAN or Geneve) back to the specific pod that initiated the connection.
Key Takeaway: Use ephemeral debugging containers that share the target’s network namespace to capture raw traffic, ensuring you see exactly what the application sees before any network policies or host-level abstractions hide the truth.