Course contentsShow
← Python
Lesson 879 of 1,06014. Concurrency and PerformanceFree lesson

Passing Data Between Processes with Message Queues

Utilize thread-safe and process-safe queues to facilitate communication and exchange data between independent processes without manual locking.

In this lesson, you will learn how to use multiprocessing.Queue to safely pass data between independent processes without the headaches of manual locking.

The Problem with Shared State

As you’ve learned, sharing memory between processes is complex because each process has its own memory space. While primitives like Value or Array allow for shared memory, they require you to manually manage synchronization (like locks) to prevent race conditions. If your processes are loosely coupled—meaning they just need to "send" a task and "receive" a result—managing these locks becomes a major performance bottleneck and a source of bugs.

Decoupling with Queues

A process-safe Queue acts as a conduit that allows one process to drop data into a buffer, which another process can then pull out. The internal implementation of multiprocessing.Queue uses pipes and locks to handle all the serialization and synchronization for you. You don't have to worry about data corruption or complex locking protocols; you simply put objects in and get objects out.

Practical Implementation

When you instantiate a Queue, it creates a buffer that is automatically shared across processes. Consider this worker-consumer pattern:

from multiprocessing import Process, Queue

def producer(q):
    q.put("Data Point A")

def consumer(q):
    item = q.get()
    print(f"Received: {item}")

if __name__ == "__main__":
    q = Queue()
    p1 = Process(target=producer, args=(q,))
    p2 = Process(target=consumer, args=(q,))
    
    p1.start()
    p2.start()
    p1.join()
    p2.join()

In this example, the producer puts a string into the queue, and the consumer pulls it out. The Queue class handles the underlying IPC (Inter-Process Communication) and ensures that the data is pickled (serialized) and reconstructed on the receiving end. This abstraction keeps your application code clean by moving the burden of thread/process safety away from your business logic and into the queue structure itself.

Key Takeaway: Use multiprocessing.Queue to facilitate clean, thread-safe, and process-safe data exchange by decoupling producers from consumers, eliminating the need for manual synchronization primitives.