Course contentsShow
← Python
Lesson 893 of 1,06014. Concurrency and PerformanceFree lesson

Using Asyncio Queues for Producer-Consumer Patterns

Coordinate data flow between multiple coroutines using thread-safe asynchronous queues to manage work distribution effectively.

In this lesson, you will learn how to decouple task generation from task execution using asyncio.Queue to build scalable, non-blocking producer-consumer pipelines.

The Producer-Consumer Pattern

In asynchronous programming, you often have a "Producer"—a coroutine that fetches or generates data—and a "Consumer"—a coroutine that processes that data. If you couple them directly (where the producer calls the processing function), the producer stalls until the processing is finished.

By using an asyncio.Queue, you create a buffer that acts as a shock absorber. The producer pushes items into the queue and immediately returns to generating more data, while one or more consumers pull items from the queue at their own pace.

Implementing the Pattern

The queue handles the synchronization internally, so you don't need to worry about race conditions between your concurrent coroutines.

import asyncio

async def producer(queue, n):
    for i in range(n):
        await asyncio.sleep(0.1)  # Simulate I/O work
        await queue.put(f"item-{i}")
        print(f"Produced item-{i}")
    await queue.put(None)  # Signal completion

async def consumer(queue):
    while True:
        item = await queue.get()
        if item is None:
            break
        print(f"Consumed {item}")
        queue.task_done()

async def main():
    queue = asyncio.Queue(maxsize=10)
    # Run producer and consumer concurrently
    await asyncio.gather(producer(queue, 5), consumer(queue))

asyncio.run(main())

Managing Flow Control

Note the maxsize parameter in asyncio.Queue. By setting a limit, you implement backpressure; if the queue fills up, the producer’s await queue.put() will pause automatically until a consumer removes an item. This prevents your program from consuming infinite memory if the producer is significantly faster than the consumer.

When the work is finished, it is common practice to send a "sentinel value" (like None) into the queue to signal the consumer to shut down gracefully. This ensures all tasks are processed before the program exits.

Key Takeaway: Asynchronous queues decouple producers from consumers, providing a thread-safe, non-blocking mechanism to manage data flow and apply backpressure in concurrent systems.