Implementing Adaptive Cache Limits

In this lesson, you will learn how to dynamically scale your in-memory cache limits based on real-time system memory availability to prevent process out of memory crashes.

The Problem with Static Limits

When you hard-code a cache limit (e.g., MAX_ENTRIES = 10000), you assume the environment's available memory is always constant. In reality, Node.js applications often share resources with other processes or fluctuate in memory usage during spikes. A static cache can either waste memory when under-utilized or cause an OOM (Out of Memory) error when the system is under pressure.

Implementing Adaptive Feedback

To create an adaptive limit, we need to monitor the process.memoryUsage() object. Specifically, we look at rss (Resident Set Size) or heapUsed. We can implement a "pressure-relief" valve that adjusts the maximum number of items in our cache based on how close we are to the system's memory ceiling.

Here is a simplified approach using an interval-based check:

const cache = new Map();
let MAX_CACHE_SIZE = 5000; // Starting limit
const MEMORY_THRESHOLD = 0.8; // 80% of allowed memory
const TOTAL_ALLOWED_MEMORY = 512 * 1024 * 1024; // 512MB limit

setInterval(() => {
  const { heapUsed } = process.memoryUsage();
  const usagePercentage = heapUsed / TOTAL_ALLOWED_MEMORY;

  if (usagePercentage > MEMORY_THRESHOLD) {
    // Aggressively shrink limit to relieve pressure
    MAX_CACHE_SIZE = Math.floor(MAX_CACHE_SIZE * 0.9);
  } else if (usagePercentage < 0.5) {
    // Gradually increase limit if memory is plentiful
    MAX_CACHE_SIZE = Math.min(MAX_CACHE_SIZE + 100, 10000);
  }

  // Evict items if we are over the new limit
  while (cache.size > MAX_CACHE_SIZE) {
    const firstKey = cache.keys().next().value;
    cache.delete(firstKey);
  }
}, 5000);

Why This Works

By observing memory pressure, your cache becomes a "good citizen" of the Node.js process. When the garbage collector works harder or the event loop slows down due to memory pressure, our adaptive mechanism reduces the memory footprint of the cache before the system forces a crash. This creates a self-healing loop where the cache expands during idle periods and contracts during high-throughput peaks.

Key Takeaway: By binding your cache capacity to process.memoryUsage() and dynamically adjusting the eviction threshold, you ensure your application remains stable even under unpredictable memory conditions.