Robots.txt Validator
Validate your website's robots.txt file
Frequently Asked Questions
What is robots.txt?
The robots.txt is a text file located at the root of your website that tells search engine bots which pages can or cannot be crawled. It follows the Robots Exclusion Protocol and is read by crawlers like Googlebot, Bingbot and others before crawling your site.
Why should I validate my robots.txt?
A misconfigured robots.txt can accidentally block important pages from your site, preventing Google from indexing them. It can also allow crawling of sensitive pages. Validation identifies syntax errors, conflicting rules and common issues.
What does 'Disallow: /' mean?
The 'Disallow: /' directive under 'User-agent: *' blocks ALL crawlers from crawling any page on your site. This is very restrictive and usually not recommended unless you deliberately want to prevent indexing of the entire site.
Do I need a Sitemap directive in robots.txt?
While not mandatory, including the Sitemap directive in robots.txt is a good SEO practice. It helps search engines find your sitemap.xml automatically, making it easier to discover and index all pages on your site.
What's the difference between Allow and Disallow?
Disallow blocks crawler access to a specific path, while Allow explicitly permits access. Allow is useful when you Disallow an entire directory but want to allow access to specific subpages. For example: Disallow: /admin/ with Allow: /admin/public/.