In this lesson, you will learn how to distinguish between the deceptive signals of system load averages and the objective reality of application throughput during peak traffic.
The Load Average Myth
In Linux, load average is a measure of demand, not performance. It represents the number of processes in the "runnable" state (actively using CPU or waiting for it) plus those in an "uninterruptible" sleep state (typically waiting for disk I/O).
As your system enters peak traffic, load averages often climb linearly. However, a high load does not necessarily imply that your application is processing requests efficiently. You might have a massive load average because of a blocked I/O thread, while your application's throughput—the number of transactions completed per second—flatlines or drops because the CPUs are stalled.
Throughput as a Business Metric
Throughput measures the actual work finished by your application. While load tells you how many processes are "in line," throughput tells you how many "customers" actually left the building. During peak periods, throughput is the ultimate arbiter of health.
Consider a scenario where you are monitoring an eBPF-based latency tracer. You might observe:
- High Load: Many processes stuck waiting for
io_submit. - Declining Throughput: The kernel is spending more time managing context switches and lock contention (which we covered in previous lessons) than executing your application's business logic.
If you rely solely on load, you might scale out your infrastructure thinking you are "out of CPU," when in reality, you are suffering from a bottleneck elsewhere, such as memory bus contention or lock-induced serial execution.
To diagnose this, don't look at uptime or top. Instead, instrument your application to report completed operations. Use bpftrace to verify the actual execution time of critical paths:
# Tracing application work completions (conceptual)
bpftrace -e 'uprobe:/usr/bin/app:process_request_complete { @start = nsecs; }'
If your load is climbing but your @start frequency is slowing down, your throughput is degrading regardless of what the system scheduler reports.
Key Takeaway: Load averages track system demand and queue length, but throughput tracks productive capacity; always optimize for the latter during peak load to avoid scaling the wrong bottleneck.