Course contentsShow
← System Design
Lesson 1833 of 1,91944. Real-World System: Web CrawlerFree lesson

Content Type Detection

Identifying MIME types, filtering non-HTML content, and deciding what to parse vs ignore during crawling.

Content Type Detection

You'll learn how crawlers identify and filter content types to avoid wasting resources on unparsable data.

Why Content Type Matters

When your crawler fetches a URL, it doesn't automatically know whether it's receiving an HTML page, a PDF, an image, or a video file. Attempting to parse a 500MB video as HTML would be catastrophic for performance. Content type detection lets you filter out non-HTML content early and decide what's worth processing.

MIME Types: The Web's Content Labels

Every HTTP response includes a Content-Type header that declares its MIME type (Multipurpose Internet Mail Extensions). Common examples:

  • text/html – HTML page ✅
  • application/pdf – PDF document
  • image/jpeg – JPEG image ❌
  • video/mp4 – Video file ❌
  • application/json – JSON data

Think of MIME types like product labels in a warehouse: they tell you what's inside before you open the box.

Filtering Strategy

Your crawler should check the Content-Type header before downloading the full response body. If it's not HTML (or occasionally XML/plain text), you can:

  1. Skip entirely – Ignore images, videos, executables
  2. Extract links only – Some PDFs contain URLs worth discovering
  3. Store metadata – Record that the URL exists but don't parse it

This decision happens after the HTTP request but before committing expensive parsing resources.

Parse vs Ignore Decision

Not all HTML is equal. You might ignore:

  • HTML pages marked noindex in robots meta tags
  • Login pages or forms (no useful links)
  • Dynamically generated content requiring JavaScript execution

Your filter acts like a bouncer: let valuable HTML through, reject the rest.

Key Takeaway: Content type detection via MIME types lets your crawler filter out non-HTML resources early, preserving bandwidth and CPU for parsable pages that actually contain discoverable links.