Meta Robots Tag
You'll learn how to control whether search engines index your pages and follow your links using meta robots directives.
What is the Meta Robots Tag?
The meta robots tag is an HTML instruction that tells search engine crawlers how to treat a specific page. Think of it as a "Do Not Disturb" sign (or "Come On In!") for your webpage—you're telling Google, Bing, and other search engines whether they should add this page to their index and whether they should follow the links on it.
The Core Directives
The meta robots tag accepts four primary values:
index— Allow this page to appear in search results (default behavior)noindex— Don't show this page in search resultsfollow— Crawl and follow links on this page (default behavior)nofollow— Don't follow or count links on this page
How to Use It
Place the tag in your page's <head> section:
<!-- Allow indexing, follow links (default) -->
<meta name="robots" content="index, follow">
<!-- Prevent indexing, but follow links -->
<meta name="robots" content="noindex, follow">
<!-- Allow indexing, but don't follow links -->
<meta name="robots" content="index, nofollow">
<!-- Block everything -->
<meta name="robots" content="noindex, nofollow">
You can target specific crawlers too:
<meta name="googlebot" content="noindex">
When to Use Each Directive
Use noindex for private pages, thank-you pages, or duplicate content you don't want in search results. Use nofollow when you don't want to pass link authority or have untrusted user-generated links.
Key Takeaway: The meta robots tag gives you fine-grained control over search engine indexing and link-following behavior on a per-page basis, helping you manage what appears in search results.