Optimizing Context Switching in Executors

In this lesson, you will learn how to tune executor heuristics to minimize the overhead of task scheduling and context switching in high-throughput environments.

The Cost of the "Ping-Pong" Effect

In a custom executor, a context switch occurs when the scheduler moves from running one Future to another. Every time a task yields control or is woken, the CPU must save registers, update stack pointers, and potentially migrate cache lines across cores. In high-throughput systems, if your executor context-switches too aggressively—for instance, by waking a task immediately upon every Waker::wake() call—you can trigger "thrashing," where the CPU spends more time moving state between tasks than actually executing them.

Batching Wakeups

To mitigate this, sophisticated executors employ wakeup batching. Instead of immediately re-queueing a task the moment it signals readiness, the executor can introduce a tiny "coalescing window." By checking the run queue only after a brief period or once a certain number of events have accumulated, you allow multiple tasks to become ready, significantly reducing the frequency of context switches.

Consider this heuristic adjustment in your poll loop:

// Heuristic: Don't switch tasks if the current one has more work 
// and we are under a specific throughput threshold.
loop {
    if let Some(task) = run_queue.pop() {
        if task.should_continue() {
            // Keep executing the current task to preserve cache locality
            task.poll(); 
        } else {
            // Switch context only when necessary
            switch_to(task);
        }
    } else {
        // Only block the thread if the queue is empty
        wait_for_io();
    }
}

Affinities and Locality

Another strategy involves thread-local run queues. Instead of a single global queue that triggers cross-core synchronization (and expensive cache invalidations), keep per-thread local queues. When a task is woken, it is pushed to the local queue of the thread that generated the wakeup. This keeps the task’s data in the L1/L2 cache of the current core, making the "switch" a local operation rather than a system-wide one.

By prioritizing task locality and delaying context switches via heuristics, you keep your CPU pipelines hot and minimize the architectural penalty of asynchronous task switching.

Key Takeaway: Minimize context switching costs by implementing wakeup batching and enforcing task affinity to keep cache state local to the executing thread.