Robots.txt Fundamentals
What you'll learn: How to use the robots.txt file to control which parts of your website search engine crawlers can access.
What is Robots.txt?
The robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers (like Googlebot) which pages or sections they're allowed to visit. Think of it as a "keep out" sign for certain areas of your site—polite bots will respect these instructions, though it's not a security measure.
Basic Syntax
The file uses a simple pattern-based syntax with two main directives:
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
- User-agent: Specifies which crawler the rules apply to (
*means all crawlers, or you can target specific bots likeGooglebot) - Disallow: Blocks crawlers from accessing specified paths
- Allow: Explicitly permits access (useful for allowing specific files within a disallowed directory)
Where to Place It
The robots.txt file must be at your domain root: https://yourdomain.com/robots.txt. It won't work in subdirectories.
Common Patterns
User-agent: *
Disallow: /wp-admin/
Disallow: /search?
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap.xml
You can also reference your sitemap here to help crawlers discover all your pages efficiently.
Key Takeaway: The robots.txt file controls crawler access through simple Allow and Disallow rules, must live at your domain root, and acts as a guideline—not a security feature—for well-behaved search engine bots.