Patterns for Producer-Consumer Workflows

Architect systems using distributed queues to decouple service producers from consumers.

In this lesson, you will learn how to decouple your application components using distributed queues to scale service producers independently from consumers.

The Decoupling Challenge

In high-throughput systems, a service (the Producer) often needs to offload heavy tasks without waiting for them to finish. If the producer waits for the consumer, the system becomes tightly coupled and synchronous, causing bottlenecks. By introducing a distributed queue (such as Azure Service Bus, RabbitMQ, or Amazon SQS), the producer merely "drops off" a message and returns to its work immediately, while the consumer picks up the task at its own pace.

Implementing the Pattern

To implement this, we define a shared contract—typically a simple Data Transfer Object (DTO)—that both services recognize. The producer serializes the data to the queue, and the consumer acts as a background worker that polls or listens for these messages.

Consider a system where a web API handles user file uploads. Instead of processing the file inside the request, it queues a message:

// The Producer (Web API Controller)
public async Task<IActionResult> Upload(FileRequest request)
{
    var message = new FileProcessingMessage { FileId = request.Id };
    await _queueClient.SendMessageAsync(message); // Fire and forget to the queue
    return Accepted(); // Return 202 immediately
}

// The Consumer (BackgroundService)
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        var message = await _queueClient.ReceiveAsync(stoppingToken);
        await ProcessFileAsync(message); // Decoupled execution
    }
}

Why This Matters

Using a distributed queue introduces "temporal decoupling." If your consumer service goes down, the messages wait safely in the queue; they are not lost. When you bring the consumer back online—or scale it out to multiple instances to handle a spike—it processes the backlog without the producer ever knowing the system was under stress. This pattern turns a fragile chain of dependent calls into a robust, asynchronous pipeline where every service focuses on its specific role without blocking the rest of the architecture.

Key Takeaway: By using distributed queues to buffer work, you ensure that producers and consumers operate independently, allowing your system to scale dynamically and withstand service downtime.