Course contentsShow
← System Design
Lesson 1873 of 1,91944. Real-World System: Web CrawlerFree lesson

Handling Recrawls and Freshness

Scheduling periodic recrawls of important pages while maintaining politeness and managing distributed state.

Handling Recrawls and Freshness

You'll learn how to keep your crawled content up-to-date by intelligently scheduling recrawls while respecting politeness constraints and managing state across distributed workers.

The Core Challenge

Web content changes constantly—news sites update hourly, blogs publish daily, government pages rarely change. Your crawler must revisit pages periodically to detect updates, but you can't recrawl everything constantly due to bandwidth, storage, and politeness limits. The challenge is deciding when to recrawl which pages while coordinating across your distributed fleet.

Recrawl Scheduling Strategy

Priority-based scheduling: High-value or frequently-changing pages get recrawled more often. Assign each URL a "freshness priority" based on:

  • Historical change frequency (using exponential moving average)
  • Page importance (PageRank, backlink count)
  • Content type (news vs archival)
  • User engagement signals

Store recrawl timestamps and next-scheduled-crawl times in your URL frontier. When workers request URLs, include both new discoveries and due-for-recrawl URLs.

Distributed State Management

Each URL needs metadata: last_crawled_time, next_crawl_time, change_frequency_estimate. Store this in a distributed database sharded by URL hash. Workers update this state after each crawl. Use the politeness table you've already built—merge recrawl scheduling with per-host queue management so recrawls respect crawl delays.

Adaptive freshness: If a page hasn't changed in 10 consecutive crawls, exponentially back off the recrawl frequency. If it suddenly changes, increase frequency again.

Practical Coordination

The coordinator doesn't schedule every recrawl centrally. Instead, workers autonomously check: "Is this URL due for recrawl?" when processing their queues. This distributes the decision-making and avoids bottlenecks.

Key Takeaway: Effective recrawling balances freshness needs with resource constraints by prioritizing important/dynamic pages, maintaining per-URL scheduling metadata in distributed storage, and letting workers make autonomous recrawl decisions within politeness boundaries.