Crawling and Document Collection
What you'll learn: How search engines discover, fetch, and store billions of documents from the web before they can ever be indexed and searched.
The Core Idea
Before a search engine can answer queries, it needs documents to search through. Crawling is the process of systematically discovering and downloading web pages (or internal documents) so they can be indexed later.
Think of it like a librarian building a library: before cataloging books, someone must first travel around collecting them. A crawler (also called a "spider" or "bot") is that collector—it starts with a seed list of URLs, visits each page, extracts links to new pages, and repeats the process continuously.
How Crawling Works
- Seed URLs: Start with a known list (popular sites, sitemaps, user submissions)
- Fetch: Download the page content via HTTP/HTTPS
- Parse: Extract text, metadata, and outgoing links
- Store: Save the raw document in a distributed storage system
- Enqueue: Add newly discovered URLs to a queue for future crawling
- Repeat: Process the next URL in the queue
Key Challenges
- Scale: Billions of pages exist; crawlers must be massively parallel
- Politeness: Respect
robots.txt, rate-limit requests to avoid overwhelming servers - Freshness: Re-crawl pages periodically to detect updates
- Duplicate Detection: Many URLs point to identical content
- Priority: Crawl important or frequently-changing pages first
Architecture Essentials
- URL Frontier (queue): Prioritized list of URLs to visit next
- DNS Resolver: Lookup IP addresses efficiently at scale
- Fetcher Pool: Distributed workers downloading pages
- Content Store: Blob storage for raw HTML/documents
- Link Extractor: Parses pages to find new URLs
Key Takeaway: Crawling is the distributed, continuous process of discovering and collecting documents from the web—respectfully and at scale—so they can be indexed and made searchable.