Building a Thread-Safe In-Memory Queue

In this lesson, you will learn how to build a high-performance, thread-safe in-memory queue by balancing memory safety with minimal synchronization overhead.

When building a queue for massive throughput, a naive implementation using Mutex<VecDeque<T>> quickly becomes a bottleneck. Under high contention, every thread must acquire the lock, leading to "convoying," where threads spend more time waiting for the lock than processing data. To achieve high performance, we shift toward designs that minimize the critical section or eliminate it entirely using atomic primitives.

The Bounded Ring Buffer Approach

For an in-memory queue, a fixed-size ring buffer is often the most performant choice. By pre-allocating memory, we avoid the overhead of dynamic resizing and heap fragmentation during runtime. We represent the queue using a Box<[T]> and two atomic indices: head (for consumption) and tail (for production).

use std::sync::atomic::{AtomicUsize, Ordering};

pub struct RingQueue<T> {
    buffer: Box<[T]>,
    head: AtomicUsize,
    tail: AtomicUsize,
    capacity: usize,
}

Minimizing Contention

To make this thread-safe without a heavy Mutex, we use AtomicUsize with Ordering::Acquire and Ordering::Release semantics. The producer only updates the tail, and the consumer only updates the head. By using these atomic operations, multiple producers can (with additional care like compare_exchange) contend for slots, or we can enforce a Single-Producer/Single-Consumer (SPSC) model for absolute maximum throughput.

The core logic relies on wrapping indices using the modulo operator: index = (index + 1) % capacity. This ensures the queue is circular. If head == tail, the queue is empty; if (tail + 1) % capacity == head, the queue is full. By keeping the logic strictly within atomic boundaries, we prevent data races while allowing threads to operate on the structure simultaneously without stalling the entire system.

By avoiding heavy-weight locks and focusing on atomic index manipulation, we ensure that our queue stays performant even as we scale to millions of operations per second, effectively turning our in-memory data structure into a high-speed pipeline for concurrent tasks.

Key Takeaway: Build thread-safe queues by replacing heavy Mutexes with atomic index tracking on a pre-allocated ring buffer to eliminate lock contention in the hot path.