Robots.txt Generator

Write a correct robots.txt without accidentally hiding your site from search.

Who can crawlAllow everythingDisallow selected pathsBlock everything (staging sites only)

Google Search (Googlebot)

Bing Search (Bingbot)

ChatGPT search (OAI-SearchBot)

Perplexity search (PerplexityBot)

Claude search (Claude-SearchBot)

ChatGPT user requests (ChatGPT-User)

Claude user requests (Claude-User)

Perplexity user requests (Perplexity-User)

OpenAI training (GPTBot)

Anthropic training (ClaudeBot)

Google AI training (Google-Extended)

Apple AI training (Applebot-Extended)

Common Crawl (CCBot)

Search crawlers find your pages for search results. Training crawlers collect data for AI models; allowing or blocking them is your choice.

Keep out

Extra paths to keep outOne path per line, each starting with /. Up to 20.

Paths to allow anywayUp to 10.

Bing crawl delay in seconds (optional)Google ignores this.

Sitemaps

Sitemap URLsFull URLs, one per line. Up to 5.

Server validation is temporarily unavailable. You can still edit, copy and download this draft in your browser.

robots.txt

1User-agent: *2Disallow: /admin/3Disallow: /login/4Disallow: /cart/5Disallow: /checkout/6Disallow: /account/7

Preview only. Review it with whoever manages your site before replacing your live robots.txt. A mistake can block intended crawling. Robots.txt does not protect private content.

How this check works

Everything runs in your browser. Nothing is sent until you choose to check the result.

Use the robots.txt checker to lint the file before you publish it.

Search outcomes vary. Rankings, indexing, traffic and inclusion in AI answers are never guaranteed.

Want this checked across your whole site?

Explore the early-access workflow for scheduled checks, reviewed findings and approval before publication.

Join the waitlist View the audit example