URL Normalization Basics
You'll learn how to canonicalize URLs so your crawler doesn't fetch the same page multiple times under different variations.
Why Normalization Matters
Web servers often serve identical content at multiple URLs. Without normalization, your crawler might visit http://example.com, HTTP://EXAMPLE.COM/, and http://example.com:80/index.html as three separate pages—wasting bandwidth, storage, and time. URL normalization converts all equivalent URLs into a single canonical form before adding them to the frontier or checking if they've already been crawled.
Core Normalization Rules
Protocol Standardization
Convert protocols to lowercase: HTTP:// → http://
Domain Case
Lowercase the hostname (DNS is case-insensitive): Example.COM → example.com
Default Ports
Remove standard ports: http://site.com:80/ → http://site.com/ and https://site.com:443/ → https://site.com/
Trailing Slashes
Decide on a consistent policy—typically, keep trailing slashes for directories, remove for files: http://site.com/path/ is canonical.
Path Normalization
Resolve . and .. segments, decode unreserved percent-encoded characters (%7E → ~), and sort query parameters alphabetically if order doesn't matter.
Before and After
- Before:
HTTP://Example.com:80/Path/../Page.html?b=2&a=1 - After:
http://example.com/page.html?a=1&b=2
This canonical form is what you store in your deduplication system (like a bloom filter or visited URL database) and compare against before queuing new URLs.
Key Takeaway:
URL normalization transforms equivalent URL variations into a single canonical form, preventing duplicate crawls and enabling accurate deduplication across your distributed crawler system.