Course contentsShow
← Web Development
Lesson 1874 of 2,98234. Metadata and SEOFree lesson

Common Robots.txt Directives

Use User-agent, Disallow, Allow, and Crawl-delay to control search engine crawler behavior.

Common Robots.txt Directives

You'll learn how to use the four essential robots.txt directives to tell search engine crawlers which parts of your site they can and cannot access.

What Are Robots.txt Directives?

In the previous lesson, you learned what robots.txt is. Now let's explore the actual instructions you write inside it. Think of directives as commands in a conversation with web crawlers—you're telling them "You can go here" or "Please stay out of this area" or "Slow down a bit, please."

The Four Core Directives

User-agent

This specifies which crawler your rules apply to. It's like addressing a letter:

User-agent: Googlebot
User-agent: *

The asterisk (*) means "all crawlers." Specific names target individual bots (Googlebot, Bingbot, etc.).

Disallow

This tells crawlers which paths to avoid:

User-agent: *
Disallow: /admin/
Disallow: /private/

This blocks all crawlers from accessing anything starting with /admin/ or /private/.

Allow

This explicitly permits access to paths that might otherwise be blocked. Useful for making exceptions:

User-agent: *
Disallow: /admin/
Allow: /admin/public-stats

This blocks /admin/ but allows /admin/public-stats.

Crawl-delay

This requests crawlers wait a specified number of seconds between requests:

User-agent: *
Crawl-delay: 10

This asks bots to wait 10 seconds between page requests, reducing server load. Note: Not all crawlers respect this directive (Google ignores it, preferring Search Console settings).

Putting It Together

User-agent: *
Disallow: /checkout/
Allow: /checkout/success
Crawl-delay: 5

Key Takeaway: Use User-agent to target specific crawlers, Disallow and Allow to control access to paths, and Crawl-delay to manage crawl speed—these four directives give you fine-grained control over crawler behavior on your site.