Common Robots.txt Directives
You'll learn how to use the four essential robots.txt directives to tell search engine crawlers which parts of your site they can and cannot access.
What Are Robots.txt Directives?
In the previous lesson, you learned what robots.txt is. Now let's explore the actual instructions you write inside it. Think of directives as commands in a conversation with web crawlers—you're telling them "You can go here" or "Please stay out of this area" or "Slow down a bit, please."
The Four Core Directives
User-agent
This specifies which crawler your rules apply to. It's like addressing a letter:
User-agent: Googlebot
User-agent: *
The asterisk (*) means "all crawlers." Specific names target individual bots (Googlebot, Bingbot, etc.).
Disallow
This tells crawlers which paths to avoid:
User-agent: *
Disallow: /admin/
Disallow: /private/
This blocks all crawlers from accessing anything starting with /admin/ or /private/.
Allow
This explicitly permits access to paths that might otherwise be blocked. Useful for making exceptions:
User-agent: *
Disallow: /admin/
Allow: /admin/public-stats
This blocks /admin/ but allows /admin/public-stats.
Crawl-delay
This requests crawlers wait a specified number of seconds between requests:
User-agent: *
Crawl-delay: 10
This asks bots to wait 10 seconds between page requests, reducing server load. Note: Not all crawlers respect this directive (Google ignores it, preferring Search Console settings).
Putting It Together
User-agent: *
Disallow: /checkout/
Allow: /checkout/success
Crawl-delay: 5
Key Takeaway: Use User-agent to target specific crawlers, Disallow and Allow to control access to paths, and Crawl-delay to manage crawl speed—these four directives give you fine-grained control over crawler behavior on your site.