Build a robots.txt that doesn't lock out AI search.
Generate a robots.txt with AI crawlers handled properly: training and search bots as separate choices, because blocking the wrong one costs AI answers.
Used for the Sitemap line. Leave blank to omit it.
Who can crawl you
Paths to keep out
Applies to every group above. robots.txt is a request, not access control — anything that must stay private needs authentication.
Seconds between requests. Google ignores it; Bing and others honour it. Leave blank unless a crawler is overloading you.
Appended verbatim. Optional.
Your robots.txt
# AI search crawlers
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# On-demand fetchers
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# Traditional search
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# Everything else
User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /apiSave this as robots.txt at your domain root, served as text/plain. It applies only to that exact host and protocol — a subdomain needs its own. Nothing you type here leaves your browser.
- Training crawlers and search crawlers as separate choices, not one switch
- Warns you if you block the agents that put you in AI answers
- Sitemap line, common disallow paths, and crawl-delay if you need it
Free, no signup, no rate limit. Nothing you check is stored.
Questions about this tool
- Should I block AI crawlers in robots.txt?
- Rarely all of them. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended) is a legitimate choice if you don't want your writing used to train models. Blocking the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) removes you from AI answers entirely, which almost nobody intends.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot gathers training data. OAI-SearchBot indexes for ChatGPT's search results. They are separate user agents with separate effects, and confusing them is the most common self-inflicted mistake in this whole field.
- Does robots.txt actually stop crawlers?
- It is a published request that well-behaved crawlers honour, not access control. It doesn't authenticate anything, and listing a path under Disallow announces that the path exists. Anything that must stay private needs a login.
- Where does robots.txt go?
- At your domain root, served as text/plain. It applies to that exact host and protocol only, so subdomains and http vs https each need their own.
- Why isn't my robots.txt being respected?
- Check your CDN. Cloudflare and others can block AI crawlers at the edge regardless of what your file permits, and some enable it by default. Our AI Crawler Checker makes a real request as each agent to catch exactly that.
Other free tools
Ready to become the answer?
Free scan in ten seconds. $99 to fix it — refunded in full if we don't.