Pipeline Parallelism in Inference
You'll learn how to split a large model across devices vertically by layer and pipeline inference requests to keep all devices busy simultaneously.
The Core Concept
Pipeline parallelism divides your model's layers across multiple devices sequentially—each device owns a subset of consecutive layers. When processing multiple requests, you create an assembly line: while device 1 processes request B's first layers, device 2 simultaneously handles request A's middle layers, and device 3 works on an earlier request's final layers.
Think of it like an automotive assembly line: station 1 installs the engine, station 2 adds the body, station 3 attaches wheels. Each station works on a different car simultaneously, maximizing throughput.
How It Works in Inference
Unlike tensor parallelism (which you learned splits layers horizontally across devices), pipeline parallelism splits the model vertically:
- Device 1: Layers 0-10
- Device 2: Layers 11-20
- Device 3: Layers 21-32
For a single request, data flows sequentially through devices. But with micro-batching, you overlap multiple requests. While request 1 moves to device 2, device 1 starts processing request 2. This overlapping fills "pipeline bubbles" and increases hardware utilization.
Challenges
Pipeline parallelism introduces pipeline bubbles—idle time when devices wait for data. The first and last devices experience more idle time. You need sufficient batch depth (multiple concurrent requests) to keep all stages busy. Load balancing matters too: uneven layer distribution creates bottlenecks.
This approach works well with continuous batching (lesson 2989), where new requests continuously fill the pipeline, minimizing bubbles.
Key Takeaway: Pipeline parallelism distributes model layers sequentially across devices and overlaps multiple inference requests to maximize utilization, but requires careful micro-batching to avoid idle "bubbles" between pipeline stages.