Course contentsShow
← Cloud and Infrastructure
Lesson 1195 of 2,98923. Kubernetes WorkloadsFree lesson

Indexed Jobs and Work Queues

Implement indexed completion mode for parallel processing with unique indices and work queue patterns for distributed tasks.

Indexed Jobs and Work Queues

What you'll learn: How to use indexed completion mode for parallel processing where each task needs a unique identifier, and how to implement work queue patterns for distributed task processing.

Understanding Indexed Jobs

When you run parallel Jobs, sometimes each task needs to know "which one am I?" Indexed completion mode assigns each Pod a unique index (0, 1, 2, etc.), allowing tasks to process specific data partitions or coordinate work without collisions.

Think of it like assigning workers specific rows in a spreadsheet—Worker 0 processes rows 0-99, Worker 1 processes rows 100-199, and so on. Each knows exactly what to do based on their index.

Setting Up an Indexed Job

In your Job manifest, specify:

apiVersion: batch/v1
kind: Job
metadata:
  name: indexed-processing
spec:
  completions: 5
  parallelism: 3
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: my-processor
        env:
        - name: JOB_COMPLETION_INDEX
          value: "0"  # Kubernetes injects actual index

The JOB_COMPLETION_INDEX environment variable tells each Pod its unique number.

Work Queue Pattern

For more dynamic workloads, work queues distribute tasks from a central queue (like Redis, RabbitMQ, or cloud message services). Pods pull tasks until the queue empties, enabling load balancing across workers with varying task durations.

This pattern uses parallelism without fixed completions—workers run until no work remains, then exit gracefully.

Key Takeaway: Use indexed Jobs when tasks need unique identifiers for data partitioning; use work queue patterns when tasks vary in complexity and need dynamic load distribution across parallel workers.