Course contentsShow
← Web Development
Lesson 1830 of 2,98234. Metadata and SEOFree lesson

Meta Robots Tag

Controlling search engine indexing and following with index/noindex and follow/nofollow directives.

Meta Robots Tag

You'll learn how to control whether search engines index your pages and follow your links using meta robots directives.

What is the Meta Robots Tag?

The meta robots tag is an HTML instruction that tells search engine crawlers how to treat a specific page. Think of it as a "Do Not Disturb" sign (or "Come On In!") for your webpage—you're telling Google, Bing, and other search engines whether they should add this page to their index and whether they should follow the links on it.

The Core Directives

The meta robots tag accepts four primary values:

  • index — Allow this page to appear in search results (default behavior)
  • noindex — Don't show this page in search results
  • follow — Crawl and follow links on this page (default behavior)
  • nofollow — Don't follow or count links on this page

How to Use It

Place the tag in your page's <head> section:

<!-- Allow indexing, follow links (default) -->
<meta name="robots" content="index, follow">

<!-- Prevent indexing, but follow links -->
<meta name="robots" content="noindex, follow">

<!-- Allow indexing, but don't follow links -->
<meta name="robots" content="index, nofollow">

<!-- Block everything -->
<meta name="robots" content="noindex, nofollow">

You can target specific crawlers too:

<meta name="googlebot" content="noindex">

When to Use Each Directive

Use noindex for private pages, thank-you pages, or duplicate content you don't want in search results. Use nofollow when you don't want to pass link authority or have untrusted user-generated links.

Key Takeaway: The meta robots tag gives you fine-grained control over search engine indexing and link-following behavior on a per-page basis, helping you manage what appears in search results.