Competing Consumers Pattern

In this lesson, you will learn how to boost your application's message-processing throughput by deploying multiple instances of a background service to work against a single shared queue.

Scaling Throughput with Competing Consumers

As your system grows, a single BackgroundService instance often becomes a bottleneck. Even if your code is highly optimized, the time required to process a single message—such as generating a PDF report or calling an external API—can limit your overall throughput. The Competing Consumers Pattern solves this by allowing multiple instances of the same worker to pull messages from a single queue.

The Mechanism

In this pattern, you deploy multiple instances of your worker (e.g., as multiple pods in Kubernetes or multiple instances in a VM Scale Set). Each instance independently polls the message broker. Because the broker is designed to handle this, it guarantees that each message is delivered to only one consumer. If Instance A picks up a message, Instance B will not receive it.

This enables horizontal scaling: if your queue depth starts rising, you can simply spin up more worker instances to drain the queue faster.

Implementation Concept

From your code's perspective, the logic remains standard BackgroundService execution. The complexity is handled by the messaging infrastructure (like RabbitMQ, Azure Service Bus, or AWS SQS). Your worker remains agnostic of its peers:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        // The messaging client handles the 'locking' or 'popping'
        // of the message so other instances don't see it.
        var message = await _messageQueue.ReceiveAsync(stoppingToken);
        
        if (message != null)
        {
            await ProcessMessageAsync(message);
            await _messageQueue.CompleteAsync(message.Id);
        }
    }
}

By decoupling the number of consumers from the volume of the workload, you achieve elastic processing power. Note that this requires your message processing to be stateless and thread-safe regarding shared resources like databases. Because you now have multiple processes hitting the same data store, ensure your data access layer handles concurrent updates appropriately.

Key Takeaway: The Competing Consumers pattern scales your background throughput horizontally by allowing multiple worker instances to concurrently process messages from a shared queue, ensuring each message is handled exactly once.