Course contentsShow
← System Design
Lesson 1605 of 1,91939. Real-World System: Image and Video HostingFree lesson

Distributed Worker Architecture

Scaling processing workers horizontally, managing job distribution, and handling failures with dead letter queues.

Distributed Worker Architecture

What you'll learn: How to scale processing workers horizontally, distribute jobs efficiently, and recover from failures using dead letter queues.

The Core Idea

After jobs land in your message queue (from our previous lesson), you need a fleet of worker machines to actually process those images and videos. A distributed worker architecture means running multiple independent worker processes across many servers, all pulling jobs from the same queue and processing them in parallel.

Think of it like a restaurant kitchen: instead of one chef handling all orders, you have multiple chefs (workers) who grab tickets (jobs) from a shared order board (queue) and work independently. If one chef gets sick, the others keep cooking.

Horizontal Scaling

Workers are stateless — they don't store data between jobs. This means you can add or remove workers at any time based on queue depth. If your queue has 10,000 pending video transcoding jobs, spin up 50 workers. When the queue empties, scale down to 5.

Job distribution happens automatically: each worker polls the queue and claims the next available job. The queue system (like SQS or RabbitMQ) ensures no two workers grab the same job through message visibility timeouts.

Handling Failures

Workers crash. Networks fail. Videos corrupt. When a worker fails mid-job, the message becomes visible again after a timeout, and another worker retries it. But what if a job keeps failing?

Dead Letter Queues (DLQ) catch poison messages. After 3-5 retry attempts, failed jobs move to a separate DLQ for manual inspection or specialized handling. This prevents one bad job from blocking the entire pipeline.

You also need idempotency: if a job runs twice (due to network issues), it shouldn't cause duplicate thumbnails or corrupted state.

Key Takeaway: Distributed workers scale processing horizontally through stateless parallelism, with message visibility ensuring safe job distribution and dead letter queues isolating failures.