Course contentsShow
← System Design
Lesson 1851 of 1,91944. Real-World System: Web CrawlerFree lesson

URL Normalization Basics

Learn how to canonicalize URLs by handling protocols, case, trailing slashes, and default ports before deduplication.

URL Normalization Basics

You'll learn how to canonicalize URLs so your crawler doesn't fetch the same page multiple times under different variations.

Why Normalization Matters

Web servers often serve identical content at multiple URLs. Without normalization, your crawler might visit http://example.com, HTTP://EXAMPLE.COM/, and http://example.com:80/index.html as three separate pages—wasting bandwidth, storage, and time. URL normalization converts all equivalent URLs into a single canonical form before adding them to the frontier or checking if they've already been crawled.

Core Normalization Rules

Protocol Standardization

Convert protocols to lowercase: HTTP:// → http://

Domain Case

Lowercase the hostname (DNS is case-insensitive): Example.COM → example.com

Default Ports

Remove standard ports: http://site.com:80/ → http://site.com/ and https://site.com:443/ → https://site.com/

Trailing Slashes

Decide on a consistent policy—typically, keep trailing slashes for directories, remove for files: http://site.com/path/ is canonical.

Path Normalization

Resolve . and .. segments, decode unreserved percent-encoded characters (%7E → ~), and sort query parameters alphabetically if order doesn't matter.

Before and After

  • Before: HTTP://Example.com:80/Path/../Page.html?b=2&a=1
  • After: http://example.com/page.html?a=1&b=2

This canonical form is what you store in your deduplication system (like a bloom filter or visited URL database) and compare against before queuing new URLs.

Key Takeaway:
URL normalization transforms equivalent URL variations into a single canonical form, preventing duplicate crawls and enabling accurate deduplication across your distributed crawler system.