Course contentsShow
Web Development
Lesson 1874 of 2,98234. Metadata and SEOPro lesson

Common Robots.txt Directives

Use User-agent, Disallow, Allow, and Crawl-delay to control search engine crawler behavior.

This lesson is for subscribers

You've completed the free preview. Subscribe to unlock every lesson in every course.