Indexed Jobs and Work Queues
What you'll learn: How to use indexed completion mode for parallel processing where each task needs a unique identifier, and how to implement work queue patterns for distributed task processing.
Understanding Indexed Jobs
When you run parallel Jobs, sometimes each task needs to know "which one am I?" Indexed completion mode assigns each Pod a unique index (0, 1, 2, etc.), allowing tasks to process specific data partitions or coordinate work without collisions.
Think of it like assigning workers specific rows in a spreadsheet—Worker 0 processes rows 0-99, Worker 1 processes rows 100-199, and so on. Each knows exactly what to do based on their index.
Setting Up an Indexed Job
In your Job manifest, specify:
apiVersion: batch/v1
kind: Job
metadata:
name: indexed-processing
spec:
completions: 5
parallelism: 3
completionMode: Indexed
template:
spec:
restartPolicy: Never
containers:
- name: worker
image: my-processor
env:
- name: JOB_COMPLETION_INDEX
value: "0" # Kubernetes injects actual index
The JOB_COMPLETION_INDEX environment variable tells each Pod its unique number.
Work Queue Pattern
For more dynamic workloads, work queues distribute tasks from a central queue (like Redis, RabbitMQ, or cloud message services). Pods pull tasks until the queue empties, enabling load balancing across workers with varying task durations.
This pattern uses parallelism without fixed completions—workers run until no work remains, then exit gracefully.
Key Takeaway: Use indexed Jobs when tasks need unique identifiers for data partitioning; use work queue patterns when tasks vary in complexity and need dynamic load distribution across parallel workers.