Course contentsShow
← System Design
Lesson 1604 of 1,91939. Real-World System: Image and Video HostingFree lesson

Message Queue for Processing Jobs

Using queues like SQS or Kafka to distribute processing tasks across worker pools and handle retries.

Message Queue for Processing Jobs

What You'll Learn

You'll learn how message queues like Amazon SQS or Apache Kafka distribute file processing tasks across worker pools, ensuring reliable, scalable, and fault-tolerant job handling for image and video operations.

The Core Problem

When users upload images or videos, processing tasks (resizing, transcoding, thumbnail generation) can't block the upload API. You need asynchronous workers to handle these jobs, but coordinating hundreds of workers processing millions of files requires a robust queuing system.

How Message Queues Work

Think of a message queue as a conveyor belt at a factory. Upload services drop "work orders" (messages) onto the belt, and worker processes pick them up independently. No direct connection needed between producers and consumers.

Key Components

Producer: Your upload service publishes a message after storing the raw file:

{ "fileId": "abc123", "operation": "transcode", "priority": "standard" }

Queue: Holds messages durably until workers retrieve them. Popular choices:

  • Amazon SQS: Managed, simple, good for decoupling
  • Apache Kafka: High-throughput, ordering guarantees, event streaming

Consumer (Worker Pool): Multiple workers poll the queue, process files, then acknowledge completion.

Handling Failures with Retries

If a worker crashes mid-transcoding, the message becomes visible again after a timeout. The queue automatically redelivers it to another worker. You configure:

  • Visibility timeout: How long a message stays invisible while processing
  • Max retries: After 3 failures, move to a dead-letter queue for investigation
  • Exponential backoff: Retry delays increase (5s, 25s, 125s...)

This prevents lost jobs and handles transient errors (network blips, out-of-memory).

Why This Matters

Queues decouple your upload API from processing capacity. You can scale workers independently, drain queues during high traffic, and guarantee every file gets processed exactly once.

Key Takeaway: Message queues provide durable, scalable task distribution with built-in retry logic, making them essential for asynchronous media processing pipelines.