Load Shedding in Async Services

In this lesson, you will learn how to protect your async services from cascading failures by implementing request rejection logic when your system hits its capacity limits.

The Problem: When Capacity Meets Queueing

In async services, a sudden burst of traffic can overwhelm your runtime. If you simply accept every incoming request, your memory usage spikes, latency degrades, and the event loop becomes congested with tasks that will likely time out before they even start. "Load shedding" is the proactive strategy of rejecting incoming work at the gate to preserve the health of the remaining system.

Implementing Load Shedding with Atomics

The most efficient way to implement load shedding is to track active tasks using an AtomicUsize. By comparing the current count against a hard limit before spawning a new task, you can drop requests instantly without pinning the CPU.

use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;

struct Service {
    active_tasks: AtomicUsize,
    max_tasks: usize,
}

impl Service {
    async fn handle_request(&self) -> Result<(), &'static str> {
        // Attempt to increment, but revert if we exceed limit
        let current = self.active_tasks.fetch_add(1, Ordering::SeqCst);
        
        if current >= self.max_tasks {
            self.active_tasks.fetch_sub(1, Ordering::SeqCst);
            return Err("Service Unavailable: Load shedding active");
        }

        // Perform work...
        
        self.active_tasks.fetch_sub(1, Ordering::SeqCst);
        Ok(())
    }
}

Why This Matters

By failing fast, you avoid the "convoy effect" where the service spends more time managing queue state and garbage collecting dropped tasks than doing actual work. While techniques like Semaphore (from tokio::sync) are common for limiting concurrency, using an atomic counter allows for a non-blocking "shedding" path. If the counter is above the limit, you return an error immediately, ensuring that the caller is informed rather than left hanging in a queue that will never be processed.

This approach honors the "Fail-Fast" principle, which is essential for distributed systems stability—it is far better to deny a request now than to time out after the user has waited five seconds.

Key Takeaway: Load shedding turns potential system crashes into manageable error states by monitoring task throughput with atomics and rejecting requests before they consume critical resources.