Content Type Detection
You'll learn how crawlers identify and filter content types to avoid wasting resources on unparsable data.
Why Content Type Matters
When your crawler fetches a URL, it doesn't automatically know whether it's receiving an HTML page, a PDF, an image, or a video file. Attempting to parse a 500MB video as HTML would be catastrophic for performance. Content type detection lets you filter out non-HTML content early and decide what's worth processing.
MIME Types: The Web's Content Labels
Every HTTP response includes a Content-Type header that declares its MIME type (Multipurpose Internet Mail Extensions). Common examples:
text/html– HTML page ✅application/pdf– PDF documentimage/jpeg– JPEG image ❌video/mp4– Video file ❌application/json– JSON data
Think of MIME types like product labels in a warehouse: they tell you what's inside before you open the box.
Filtering Strategy
Your crawler should check the Content-Type header before downloading the full response body. If it's not HTML (or occasionally XML/plain text), you can:
- Skip entirely – Ignore images, videos, executables
- Extract links only – Some PDFs contain URLs worth discovering
- Store metadata – Record that the URL exists but don't parse it
This decision happens after the HTTP request but before committing expensive parsing resources.
Parse vs Ignore Decision
Not all HTML is equal. You might ignore:
- HTML pages marked
noindexin robots meta tags - Login pages or forms (no useful links)
- Dynamically generated content requiring JavaScript execution
Your filter acts like a bouncer: let valuable HTML through, reject the rest.
Key Takeaway: Content type detection via MIME types lets your crawler filter out non-HTML resources early, preserving bandwidth and CPU for parsable pages that actually contain discoverable links.