Course contentsShow
← Python
Lesson 882 of 1,06014. Concurrency and PerformanceFree lesson

Implementing Pipes for Bidirectional Inter-Process Communication

Establish direct communication channels between two processes using pipes to facilitate low-latency data transfer for specific master-worker patterns.

Implementing Pipes for Bidirectional Inter-Process Communication

In this lesson, you will learn how to establish high-speed, direct communication channels between two processes using pipes for efficient, low-latency data exchange.

While multiprocessing.Queue is excellent for many-to-many communication, it introduces overhead because it is built upon underlying pipes and locks to ensure thread and process safety. When you have a strict master-worker pattern where two processes need to talk back and forth frequently, multiprocessing.Pipe() offers a leaner, faster alternative.

How Pipes Work

A pipe returns a pair of connection objects representing the two ends of the pipe. You can think of it like a telephone line: whatever you send() into one end is immediately available to be recv()’d at the other. Because a pipe is strictly bidirectional between two specific endpoints, it avoids the synchronization complexity required for shared queues.

Consider this minimal setup where a master sends a task to a worker and waits for a specific confirmation:

from multiprocessing import Process, Pipe

def worker(conn):
    # Receive data from the master
    msg = conn.recv()
    print(f"Worker received: {msg}")
    # Send a result back directly to the master
    conn.send("Task Complete")
    conn.close()

if __name__ == "__main__":
    parent_conn, child_conn = Pipe()
    p = Process(target=worker, args=(child_conn,))
    p.start()

    parent_conn.send("Process this data")
    print(f"Master received: {parent_conn.recv()}")
    p.join()

Why Use Pipes?

The primary advantage here is performance. Because pipes are point-to-point, they provide a lower-latency path than the more feature-rich queue objects. However, there is a trade-off: pipes are not process-safe if multiple processes attempt to write to or read from the same end simultaneously. For the master-worker pattern where each process owns one end of the connection, this is rarely an issue. By avoiding the extra management layer of a queue, you reduce the per-message latency significantly, making this ideal for high-frequency messaging between two specialized processes.

Key Takeaway: Use Pipes for low-latency, point-to-point communication between two processes when you want to minimize overhead and do not require the multi-consumer capabilities of a queue.