Dynamic Load Shedding

In this lesson, you will learn how to maintain system stability by intentionally dropping non-essential requests when your caching layer faces a critical bottleneck.

The Concept of Load Shedding

When your database or cache hits its threshold—perhaps due to a sudden traffic spike or a cache stampede—every incoming request contributes to the pressure. If you continue to process every task, the entire service risks a total crash. Dynamic Load Shedding is the defensive practice of detecting these pressure points and preemptively rejecting lower-priority tasks to preserve resources for your most vital operations.

Unlike a circuit breaker, which cuts off traffic to a failing service, load shedding is about discriminating between traffic types based on current resource utilization.

Practical Implementation

To implement this, you can wrap your cache access in a middleware layer that checks system health indicators, such as memory usage or pending request queues. If the system is under duress, you drop non-essential tasks (like background analytics or non-critical personalization) while prioritizing core requests (like authentication or purchase paths).

def get_user_data(key, is_critical):
    # Monitor system state (e.g., check memory pressure)
    if cache_is_under_load():
        if not is_critical:
            # Drop the request to save the system
            raise Exception("Load Shed: Service temporarily unavailable for background tasks")
    
    # Proceed with core request
    return cache.get(key)

Deciding What to Shed

The success of load shedding depends on how you classify traffic. You should define a hierarchy for your requests:

1. Critical: Requests essential for the core user experience (e.g., login, checkout).
2. Degraded: Requests that can be skipped without breaking functionality (e.g., "Recommended for You" lists).
3. Background: Asynchronous tasks that can wait for a retry (e.g., email notifications).

When the cache hits 90% memory capacity, your logic should trigger, dropping "Background" requests first. If the pressure persists, you move to "Degraded" requests. By sacrificing the non-essential, you keep your database and cache performing for the users who need it most.

Key Takeaway: Dynamic load shedding protects your infrastructure by sacrificing non-essential traffic when the system is under resource pressure to ensure core services remain responsive.