What are Crawler Directives? - Accuranker
Free Hub Resources
Crawler directives tell search engines how to behave when crawling and indexing your website. Examples of crawler directives are the robots meta tag, the x-robots tag and the robots.txt file.