Course contentsShow
← Web Development
Lesson 1873 of 2,98234. Metadata and SEOFree lesson

Robots.txt Fundamentals

Learn robots.txt syntax, how to allow or disallow crawlers, and where to place the file.

Robots.txt Fundamentals

What you'll learn: How to use the robots.txt file to control which parts of your website search engine crawlers can access.

What is Robots.txt?

The robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers (like Googlebot) which pages or sections they're allowed to visit. Think of it as a "keep out" sign for certain areas of your site—polite bots will respect these instructions, though it's not a security measure.

Basic Syntax

The file uses a simple pattern-based syntax with two main directives:

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
  • User-agent: Specifies which crawler the rules apply to (* means all crawlers, or you can target specific bots like Googlebot)
  • Disallow: Blocks crawlers from accessing specified paths
  • Allow: Explicitly permits access (useful for allowing specific files within a disallowed directory)

Where to Place It

The robots.txt file must be at your domain root: https://yourdomain.com/robots.txt. It won't work in subdirectories.

Common Patterns

User-agent: *
Disallow: /wp-admin/
Disallow: /search?
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml

You can also reference your sitemap here to help crawlers discover all your pages efficiently.

Key Takeaway: The robots.txt file controls crawler access through simple Allow and Disallow rules, must live at your domain root, and acts as a guideline—not a security feature—for well-behaved search engine bots.