Course contentsShow
← System Design
Lesson 1849 of 1,91944. Real-World System: Web CrawlerFree lesson

URL Frontier Persistence and Recovery

Persisting frontier state to disk or distributed storage to recover from failures without losing discovered URLs.

URL Frontier Persistence and Recovery

What You'll Learn

You'll learn how to persist the URL frontier state to disk or distributed storage so your crawler can recover from failures without losing millions of discovered URLs.

The Problem: Frontier is Critical State

Your URL frontier holds millions (or billions) of URLs waiting to be crawled. If your crawler crashes or needs to restart, losing this state means losing all discovered work — you'd have to start from scratch with just your seed URLs. That's catastrophic for large-scale crawlers.

Persistence Strategy

Checkpoint the frontier periodically to durable storage. This involves:

What to Persist

  • Front queues (priority queues with scored URLs)
  • Back queues (per-host politeness queues)
  • Politeness table (last-access timestamps per host)
  • Seen URL bloom filter or set (to avoid re-crawling)

Where to Persist

  • Local disk: Fast append-only log files or snapshots
  • Distributed storage: S3, HDFS, or database for multi-node crawlers
  • Write-ahead log (WAL): Log every frontier mutation before applying it

Recovery Process

On restart:

  1. Load the last checkpoint from storage
  2. Replay any WAL entries since that checkpoint
  3. Rebuild in-memory queues and politeness state
  4. Resume crawling from where you left off

Analogy

Think of a video game with save points. Without saving, a crash means replaying from the beginning. With periodic saves (checkpoints), you lose at most a few minutes of progress — not hours of gameplay.

Key Takeaway:
Persisting URL frontier state to durable storage enables crash recovery without losing discovered URLs, making your crawler resilient to failures at scale.