Course contentsShow
← System Design
Lesson 1829 of 1,91944. Real-World System: Web CrawlerFree lesson

URL Discovery and Extraction

Parsing HTML to extract links, handling relative vs absolute URLs, and normalizing discovered URLs.

URL Discovery and Extraction

You'll learn how a web crawler identifies, extracts, and normalizes new URLs from HTML pages to systematically discover the web.

What URL Discovery Means

Once your crawler fetches an HTML page, its work isn't done—that page is a treasure map. The HTML contains hyperlinks pointing to other pages, and extracting these links is how the crawler discovers new URLs to visit. Without this step, your crawler would be stuck visiting only the initial seed URLs.

Parsing HTML for Links

The crawler uses an HTML parser (like BeautifulSoup, jsoup, or native DOM parsers) to identify all <a> tags with href attributes, plus other link sources like <img>, <script>, and <link> tags. Think of it like reading a book and highlighting every footnote reference—those references lead you to more content.

Relative vs Absolute URLs

Not all extracted URLs are complete. A link might be:

  • Absolute: https://example.com/page2 (ready to use)
  • Relative: /page2 or ../products/item (needs the current page's base URL)

Your crawler must resolve relative URLs against the current page's URL to create valid, absolute URLs. It's like giving directions: "turn left" only makes sense if you know where someone is starting from.

URL Normalization

Before adding URLs to the queue, normalize them to avoid visiting duplicates:

  • Convert HTTP://Example.COM/Page → http://example.com/page (lowercase)
  • Remove fragments: page.html#section → page.html
  • Handle default ports: http://site.com:80/ → http://site.com/
  • Sort query parameters consistently

Key Takeaway: URL discovery involves parsing HTML to extract links, resolving relative URLs into absolute ones, and normalizing them to prevent duplicate crawling—turning each page into a roadmap for finding more content.