URL Discovery and Extraction
You'll learn how a web crawler identifies, extracts, and normalizes new URLs from HTML pages to systematically discover the web.
What URL Discovery Means
Once your crawler fetches an HTML page, its work isn't done—that page is a treasure map. The HTML contains hyperlinks pointing to other pages, and extracting these links is how the crawler discovers new URLs to visit. Without this step, your crawler would be stuck visiting only the initial seed URLs.
Parsing HTML for Links
The crawler uses an HTML parser (like BeautifulSoup, jsoup, or native DOM parsers) to identify all <a> tags with href attributes, plus other link sources like <img>, <script>, and <link> tags. Think of it like reading a book and highlighting every footnote reference—those references lead you to more content.
Relative vs Absolute URLs
Not all extracted URLs are complete. A link might be:
- Absolute:
https://example.com/page2(ready to use) - Relative:
/page2or../products/item(needs the current page's base URL)
Your crawler must resolve relative URLs against the current page's URL to create valid, absolute URLs. It's like giving directions: "turn left" only makes sense if you know where someone is starting from.
URL Normalization
Before adding URLs to the queue, normalize them to avoid visiting duplicates:
- Convert
HTTP://Example.COM/Page→http://example.com/page(lowercase) - Remove fragments:
page.html#section→page.html - Handle default ports:
http://site.com:80/→http://site.com/ - Sort query parameters consistently
Key Takeaway: URL discovery involves parsing HTML to extract links, resolving relative URLs into absolute ones, and normalizing them to prevent duplicate crawling—turning each page into a roadmap for finding more content.