Course contentsShow
← System Design
Lesson 1732 of 1,91942. Real-World System: Search and TypeaheadFree lesson

Crawling and Document Collection

How search engines discover and collect documents from the web or internal systems for indexing.

Crawling and Document Collection

What you'll learn: How search engines discover, fetch, and store billions of documents from the web before they can ever be indexed and searched.


The Core Idea

Before a search engine can answer queries, it needs documents to search through. Crawling is the process of systematically discovering and downloading web pages (or internal documents) so they can be indexed later.

Think of it like a librarian building a library: before cataloging books, someone must first travel around collecting them. A crawler (also called a "spider" or "bot") is that collector—it starts with a seed list of URLs, visits each page, extracts links to new pages, and repeats the process continuously.


How Crawling Works

  1. Seed URLs: Start with a known list (popular sites, sitemaps, user submissions)
  2. Fetch: Download the page content via HTTP/HTTPS
  3. Parse: Extract text, metadata, and outgoing links
  4. Store: Save the raw document in a distributed storage system
  5. Enqueue: Add newly discovered URLs to a queue for future crawling
  6. Repeat: Process the next URL in the queue

Key Challenges

  • Scale: Billions of pages exist; crawlers must be massively parallel
  • Politeness: Respect robots.txt, rate-limit requests to avoid overwhelming servers
  • Freshness: Re-crawl pages periodically to detect updates
  • Duplicate Detection: Many URLs point to identical content
  • Priority: Crawl important or frequently-changing pages first

Architecture Essentials

  • URL Frontier (queue): Prioritized list of URLs to visit next
  • DNS Resolver: Lookup IP addresses efficiently at scale
  • Fetcher Pool: Distributed workers downloading pages
  • Content Store: Blob storage for raw HTML/documents
  • Link Extractor: Parses pages to find new URLs

Key Takeaway: Crawling is the distributed, continuous process of discovering and collecting documents from the web—respectfully and at scale—so they can be indexed and made searchable.