In this lesson, you will learn how to tune the Linux kernel networking stack and NIC hardware buffers to eliminate silent packet loss under heavy load.
When designing for zero-packet-loss, your primary enemy is the "burstiness" of network traffic. If the incoming packet rate temporarily exceeds the kernel’s ability to process them, the ingress buffers fill up, leading to tail drops. To minimize this probability, we must optimize the journey from the physical wire to the application memory.
1. Scaling the Receive Side (RSS and RPS)
If your traffic is hitting a single CPU core, that core becomes a bottleneck while others sit idle. Use Receive Side Scaling (RSS) to distribute traffic across hardware queues, or Receive Packet Steering (RPS) if your NIC hardware is limited. This ensures that no single CPU core is overwhelmed, preventing drops during sudden traffic spikes.
2. Tuning Buffer Sizes
The kernel uses ring buffers to hold packets before they are processed by the stack. If these are too small, packets are dropped at the NIC level before the kernel even sees them. You can check your current configuration with ethtool -g <interface>. To minimize drops, increase these values to the hardware maximum:
# Check current ring buffer settings
ethtool -g eth0
# Increase RX ring buffer to hardware max (adjust based on ethtool output)
ethtool -G eth0 rx 4096
3. Mitigating SoftIRQ Backlog
Even if the hardware is tuned, the kernel's netdev_max_backlog determines how many packets can sit in the queue waiting for the CPU to process the softirq. If this queue is full, packets are discarded. Ensure this value is large enough to absorb bursts:
# View and increase the kernel backlog
sysctl net.core.netdev_max_backlog
sysctl -w net.core.netdev_max_backlog=5000
4. Avoiding Context Switching
Frequent context switching between the kernel stack and user-space applications leads to latency and drops. By using tools like taskset to pin your network-intensive processes to specific CPU cores that are also handling the NIC interrupts (the smp_affinity settings), you minimize cache misses and keep the processing pipeline hot.
Key Takeaway: Zero-packet-loss requires aligning your hardware buffer capacity with your kernel's processing throughput by scaling across CPU cores and increasing backlog limits to absorb traffic bursts.