Course contentsShow
← System Design
Lesson 1864 of 1,91944. Real-World System: Web CrawlerFree lesson

Stateless Worker Design

Designing crawler workers as stateless services that can be horizontally scaled without coordination overhead.

Stateless Worker Design

What You'll Learn

You'll learn how to design crawler workers as stateless services that can scale horizontally without coordination overhead, enabling elastic capacity and fault tolerance.

The Core Idea

In a distributed web crawler, stateless workers are processing units that don't store any persistent state locally. Each worker receives a URL, fetches it, extracts new URLs, and reports back—then immediately forgets everything about that task. All state (seen URLs, frontier queues, crawl metadata) lives in shared external storage like Redis or a database.

Think of stateless workers like servers in a busy restaurant kitchen. Each server (worker) takes an order ticket (URL), delivers the food (fetches content), and returns—but they don't remember past orders or claim ownership of specific tables. Any server can handle any ticket. If a server goes home sick, another picks up the next ticket without missing a beat.

Why Stateless Matters

Horizontal Scaling: You can add or remove workers at any time without coordination. Spin up 100 more workers during peak hours, then scale down—each new worker is immediately productive.

Fault Tolerance: If a worker crashes mid-fetch, nothing is lost permanently. The coordinator can reassign that URL to another worker. No complex state recovery needed.

Load Balancing: Any worker can process any URL from the frontier. No "sticky" assignments or worker-specific queues to manage.

Deployment Simplicity: Rolling updates are trivial—kill a worker, deploy new version, start it up. No state migration required.

The trade-off is that workers must fetch state (like DNS cache, robots.txt rules) from shared stores on each task, but proper caching strategies (covered earlier) mitigate this overhead.

Key Takeaway: Designing crawler workers as stateless services maximizes elasticity and resilience by eliminating local state dependencies, allowing workers to be added, removed, or replaced without coordination complexity.