Robots.txt Checker
See which search engines and AI crawlers your robots.txt lets in, line by line.
Checking whether this tool is available…
fieldwork.example/robots.txt
User-agent: *Disallow: /checkout/Disallow: /products/Sitemap: https://fieldwork.example/sitemap.xml
- PassSitemap is declaredCrawlers can find sitemap.xml.Pass
- Fix/products/ is blockedYour product pages cannot be crawled.Fix
- Pass/checkout/ is blockedCheckout pages stay out of search.Pass
How this check works
We fetch /robots.txt from your site once and read it the way Google documents: the most specific user-agent group applies, the longest matching rule wins, and Allow wins a tie.
Bot purposes come from each operator's published documentation, checked 4 Nov 2026.
Search outcomes vary. Rankings, indexing, traffic and inclusion in AI answers are never guaranteed.
Want this checked across your whole site?
Explore the early-access workflow for scheduled checks, reviewed findings and approval before publication.