Technical SEO
Online robots.txt tester
Analyze robots.txt rules and check whether a URL is allowed for a specific user agent.
robots.txt analyzer
robots.txt guide
What is robots.txt?
robots.txt is a file that tells crawlers which paths they may crawl on a site. It is usually published at the domain root.
Why test robots.txt?
It helps find accidental blocks, verify changes before publishing and understand which rules apply to each crawler.
Allow versus Disallow
Allow permits a path and Disallow blocks it for the relevant user-agent group. When rules conflict, the most specific match is used.
robots.txt and SEO
robots.txt affects crawling, but it is not a security mechanism and does not guarantee that a URL will stay out of search results.
Frequently asked questions
robots.txt questions
How do I know if a URL is blocked by robots.txt?
Paste the file, enter the URL and choose the user-agent you want to simulate.
What does User-agent: * mean?
It identifies a rules group that applies to any crawler without a more specific group.
What is the difference between Allow and Disallow?
Allow permits crawling a path and Disallow asks crawlers not to crawl it.
Does robots.txt stop a page appearing in Google?
Not necessarily. robots.txt mainly controls crawling and does not replace indexing directives or access protection.
Can I block the whole site?
Yes. Disallow: / in the relevant group blocks simulated crawling of every path.
What is Sitemap for?
It points crawlers to an XML sitemap location to help them discover URLs.
Is robots.txt a security system?
No. It is public and anyone can read it; it does not protect content or prevent direct requests.