Build a robots.txt that doesn't lock out AI search.
Generate a robots.txt with AI crawlers handled properly: training and search bots as separate choices, because blocking the wrong one costs AI answers.
Used for the Sitemap line. Leave blank to omit it.
Who can crawl you
Paths to keep out
Applies to every group above. robots.txt is a request, not access control — anything that must stay private needs authentication.
Seconds between requests. Google ignores it; Bing and others honour it. Leave blank unless a crawler is overloading you.
Appended verbatim. Optional.
Your robots.txt
# AI search crawlers
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# Training and AI-use controls
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# On-demand fetchers
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# Traditional search
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
# Everything else
User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /apiSave this as robots.txt at your domain root, served as text/plain. It applies only to that exact host and protocol — a subdomain needs its own. Nothing you type here leaves your browser.
- Training crawlers and search crawlers as separate choices, not one switch
- Warns you if you block the agents that put you in AI answers
- Sitemap line, common disallow paths, and crawl-delay if you need it
Free, no signup, no rate limit. Nothing you check is stored.
How to use the robots.txt generator
Enter your site URL
Used only for the Sitemap line. Leave it blank to omit that line entirely.
Decide on each crawler group
AI search, AI training, on-demand fetchers and traditional search are separate choices because they have separate consequences.
Add the paths to keep out
Admin, checkout and account areas. Remember robots.txt is a request, not access control — anything genuinely private needs authentication.
Publish at your domain root
Save it as robots.txt at the root, served as text/plain. It applies to that exact host and protocol; a subdomain needs its own.
Blocking training without blocking answers
If you want your writing kept out of model training but still want to appear in AI answers, block the training crawlers and allow the search ones. GPTBot, ClaudeBot, Google-Extended and Applebot-Extended gather training data. OAI-SearchBot, PerplexityBot and Claude-SearchBot retrieve pages to answer live questions. They are different user agents obeying different rules, and conflating them is the single most common self-inflicted visibility problem we see.
What robots.txt cannot do
It cannot keep a page out of search results — a disallowed URL can still be indexed from links pointing at it, listed without a description. Use a noindex meta tag for that, and note the page must be crawlable for noindex to be read at all. It also cannot protect anything: the file is public, and listing a sensitive path in it is an advertisement. Real protection is authentication, and always was.
Questions about this tool
- Should I block AI crawlers in robots.txt?
- Rarely all of them. Blocking training crawlers such as GPTBot and ClaudeBot is a separate choice from allowing search crawlers. Google-Extended is a control token, not a crawler. Blocking search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot can limit direct access to your public pages for search-backed answers. Decide separately from model-training access.
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot gathers training data. OAI-SearchBot indexes for ChatGPT's search results. They are separate user agents with separate effects, and confusing them is the most common self-inflicted mistake in this whole field.
- Does robots.txt actually stop crawlers?
- It is a published request that well-behaved crawlers honour, not access control. It doesn't authenticate anything, and listing a path under Disallow announces that the path exists. Anything that must stay private needs a login.
- Where does robots.txt go?
- At your domain root, served as text/plain. It applies to that exact host and protocol only, so subdomains and http vs https each need their own.
- Why isn't my robots.txt being respected?
- Check your CDN. Cloudflare and others can block AI crawlers at the edge regardless of what your file permits, and some enable it by default. Our AI Crawler Checker probes with crawler user-agent strings, but only verified bot events in your CDN or server logs can confirm whether the real platform crawler was blocked.
Other free tools
Put this resource into practice
Launch today.
Keep growing tomorrow.
Give your product a home, then explore the tools that help you get discovered.