Course contentsShow
← System Design
Lesson 1835 of 1,91944. Real-World System: Web CrawlerFree lesson

Crawl Freshness Requirements

Balancing recrawl frequency vs resource usage based on content type, update frequency, and priority.

Crawl Freshness Requirements

What you'll learn: How to decide when and how often to recrawl pages based on their content characteristics and system constraints.

The Core Challenge

Not all web pages need the same attention from your crawler. A news homepage changes every hour, while a university's history page might stay unchanged for years. Crawl freshness is about keeping your index up-to-date while respecting resource limits—you can't recrawl the entire web constantly.

The fundamental trade-off: recrawl too aggressively and you waste bandwidth, CPU, and storage; recrawl too conservatively and users see stale content.

Factors That Drive Recrawl Frequency

Content Type & Update Patterns:

  • News sites, social feeds: hourly or more
  • Blogs, product pages: daily
  • Documentation, legal pages: weekly or monthly
  • Historical archives: rarely

Priority Signals:

  • Page authority (high PageRank = more frequent)
  • User traffic (popular pages deserve freshness)
  • Business importance (homepage > deep archive)
  • Historical change rate (pages that changed often likely will again)

Resource Constraints:

  • Bandwidth budget per domain
  • Crawler fleet capacity
  • Storage for delta detection
  • Politeness intervals (respecting robots.txt delays)

The Balancing Act

Think of it like grocery shopping: milk expires quickly (recrawl often), canned goods last months (check occasionally). Your crawler maintains a priority queue where each URL has a "next crawl time" calculated from these factors.

A typical formula weighs: recrawl_priority = f(page_importance, historical_change_rate, time_since_last_crawl, content_type)

Key Takeaway: Crawl freshness is about intelligently allocating finite crawling resources to maximize index quality—frequently recrawling high-value, volatile content while minimizing waste on static pages.