The Network Administrator’s Toolkit

Select the appropriate combination of native tools for full stack visibility.

The Network Administrator’s Toolkit: Orchestrating Stack Visibility

In this lesson, you will learn how to synthesize native Linux diagnostic tools to gain comprehensive visibility into the kernel network stack, moving beyond simple interface monitoring to full path analysis.

As an administrator, relying on a single utility is rarely sufficient when diagnosing complex latency or packet loss issues. The "Toolkit" approach relies on layering specific kernel-native utilities that correlate state across different layers of the networking model. To achieve full-stack visibility, you must learn to bridge the gap between process-level metrics and kernel-level packet processing.

The Diagnostic Triad

To see the full picture, you must combine three specific categories of tools:

Observability (ss and nstat): Use ss -tie to extract real-time TCP state information, including congestion windows, retransmission timeouts, and smoothed round-trip times. Supplement this with nstat to track kernel-level packet counters, which reveal if drops are occurring before they reach your application.
Path Analysis (ip route get and tc): Before debugging a packet, you must verify the kernel’s forwarding decision. Use ip route get <destination> to see exactly which interface and routing table path the kernel selects. Combine this with tc -s qdisc show dev <interface> to inspect queuing disciplines, identifying where packets might be buffered or dropped due to strict QoS policies.
Flow Correlation (conntrack): Since you are managing large-scale connection tracking, use conntrack -L to inspect the state of the kernel’s connection table. This allows you to verify if traffic is being rejected due to state mismatches, which often occur when asymmetric routing or strict firewall policies interfere with the stack.

Example: Correlating a Drop If you suspect a service is dropping packets, first identify the socket state and the kernel’s internal count:

# Check socket-specific TCP health
ss -tie state established '( dport = :80 )'

# Check kernel global packet counters for drops
nstat -az | grep -i drop

If these tools show a discrepancy—such as high TcpExtListenDrops but low socket-level errors—you know the bottleneck exists in the kernel’s accept queue rather than the application itself. By layering these commands, you move from "guessing" to precise path-based diagnosis.

Key Takeaway: Full-stack visibility is achieved by layering socket-level telemetry, kernel-level packet counters, and routing-table inspection to correlate state across the entire networking path.