In this lesson, you will learn how to decouple task generation from task execution using asyncio.Queue to build scalable, non-blocking producer-consumer pipelines.
The Producer-Consumer Pattern
In asynchronous programming, you often have a "Producer"—a coroutine that fetches or generates data—and a "Consumer"—a coroutine that processes that data. If you couple them directly (where the producer calls the processing function), the producer stalls until the processing is finished.
By using an asyncio.Queue, you create a buffer that acts as a shock absorber. The producer pushes items into the queue and immediately returns to generating more data, while one or more consumers pull items from the queue at their own pace.
Implementing the Pattern
The queue handles the synchronization internally, so you don't need to worry about race conditions between your concurrent coroutines.
import asyncio
async def producer(queue, n):
for i in range(n):
await asyncio.sleep(0.1) # Simulate I/O work
await queue.put(f"item-{i}")
print(f"Produced item-{i}")
await queue.put(None) # Signal completion
async def consumer(queue):
while True:
item = await queue.get()
if item is None:
break
print(f"Consumed {item}")
queue.task_done()
async def main():
queue = asyncio.Queue(maxsize=10)
# Run producer and consumer concurrently
await asyncio.gather(producer(queue, 5), consumer(queue))
asyncio.run(main())
Managing Flow Control
Note the maxsize parameter in asyncio.Queue. By setting a limit, you implement backpressure; if the queue fills up, the producer’s await queue.put() will pause automatically until a consumer removes an item. This prevents your program from consuming infinite memory if the producer is significantly faster than the consumer.
When the work is finished, it is common practice to send a "sentinel value" (like None) into the queue to signal the consumer to shut down gracefully. This ensures all tasks are processed before the program exits.
Key Takeaway: Asynchronous queues decouple producers from consumers, providing a thread-safe, non-blocking mechanism to manage data flow and apply backpressure in concurrent systems.