Which AI crawlers to allow, and how
Every AI crawler that matters, what each one actually does, and the robots.txt block to copy. Blocking the wrong one removes you from answers you wanted to be in.
Most AI companies run more than one crawler, and they do different jobs. Blocking the training crawler while hoping to appear in that company's search-backed answers is the single most common self-inflicted wound in this field — and it is usually a decision someone made in 2023 and never revisited.
The crawlers, and what each one does
| User agent | Operator | What it does | Blocking it means |
|---|---|---|---|
| GPTBot | OpenAI | Crawls to train future models | Less likely to be known by default |
| OAI-SearchBot | OpenAI | Indexes for ChatGPT search results | You can't appear in search-backed answers |
| ChatGPT-User | OpenAI | Fetches a page a user asked about | Users can't pull your page into a chat |
| PerplexityBot | Perplexity | Indexes for Perplexity answers | Removed from general Perplexity answers |
| Perplexity-User | Perplexity | Fetches a page on user request | Users can't pull your page in directly |
| ClaudeBot | Anthropic | Crawls for training | Less likely to be known by Claude |
| Claude-User | Anthropic | Fetches on user request | Users can't pull your page into Claude |
| Claude-SearchBot | Anthropic | Indexes for Claude's search | Removed from Claude's search results |
| Google-Extended | A control token, not a crawler — governs Gemini and grounding | Content excluded from Gemini; Search rank unaffected | |
| Googlebot | Ordinary Search crawling, which AI Overviews are built on | Removed from Google entirely | |
| Applebot-Extended | Apple | Control token for Apple Intelligence training | Excluded from Apple's models |
| Bingbot | Microsoft | Search index behind Copilot | Removed from Bing and Copilot |
The block to copy
If you want to be found in AI answers, this is the whole configuration. Naming each agent explicitly is not required — a permissive default already allows them — but it states intent, and it means the next person to touch this file has to make a deliberate choice rather than an accidental one.
# --- OpenAI ---
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# --- Perplexity ---
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# --- Anthropic ---
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# --- Google ---
User-agent: Google-Extended
Allow: /
# --- Microsoft / Apple ---
User-agent: Bingbot
Allow: /
User-agent: Applebot-Extended
Allow: /
# --- Everything else ---
User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api
Sitemap: https://yoursite.com/sitemap.xmlShould you block the training crawlers?
This is a real decision with a real trade-off, and the honest answer depends on what your content is worth to you.
| Allow training crawlers | Block them |
|---|---|
| Your business can be known without a live search | Your writing isn't used to train a competitor's model |
| Answers about you work offline from retrieval | You keep a clearer claim if licensing ever matters |
| Costs you nothing you were selling | You lose the default-knowledge path entirely |
For most businesses the calculus is easy: you are not in the content-licensing business, and being known is the entire point. For publishers whose archive is the product, blocking training while allowing search is a coherent position — and it is exactly why the two crawlers are separate.
How to check what you have now
- 1
Read your own file
Open yoursite.com/robots.txt in a browser. Read every Disallow. Plenty of sites blocked AI crawlers on general principle in 2023 and never revisited it.
- 2
Check the wildcard rules
A
User-agent: *block with a broad Disallow applies to every crawler that has no rule of its own. That is how sites block AI crawlers without ever naming one. - 3
Confirm the file is actually served
A 404 page returning 200 with HTML in it is worse than no file — crawlers may treat unparseable content unpredictably.
- 4
Check your CDN and firewall too
robots.txt is a request, not a wall. Cloudflare, Vercel and AWS WAF can all block AI crawlers at the edge regardless of what your file says, and several enable this by default.
# What your robots.txt actually says
curl -s https://yoursite.com/robots.txt
# Does the site answer a crawler at all? (a 403 here is your answer)
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" \
https://yoursite.com/robots.txt is not a security control
It is a published request that well-behaved crawlers honour. It does not authenticate, does not enforce, and listing a path under Disallow announces that the path exists. Anything that must not be read needs authentication, not a line in a text file.
Once crawlers can reach you, the next question is whether what they find says anything useful — which is why businesses stay invisible even with a permissive file.
Frequently asked questions
- What is GPTBot?
- OpenAI's crawler for gathering training data. It is separate from OAI-SearchBot, which indexes for ChatGPT's search results, and from ChatGPT-User, which fetches a page when a user asks about it directly.
- Should I block GPTBot?
- Only if you don't want your content used for model training and accept being less known by default. It is a legitimate choice for publishers. For most businesses, being known is the point — and blocking it does nothing to protect you commercially.
- Does blocking GPTBot remove me from ChatGPT?
- No. Search-backed answers come from OAI-SearchBot. Blocking GPTBot only reduces the chance the model learns about you during training. Many sites block one while intending the other.
- Does Google-Extended affect my Google rankings?
- Google states it does not. It governs use in Gemini and grounded generative answers, not Search ranking. AI Overviews eligibility follows ordinary Search snippet settings instead.
- Do AI crawlers actually obey robots.txt?
- The major operators publish that they do, and their crawlers are identifiable by user agent and published IP ranges. It is a voluntary protocol, so it is not a guarantee — treat it as a request, never as access control.
- Where should robots.txt live?
- At your domain root — https://yoursite.com/robots.txt — served as text/plain. It applies only to that exact host and protocol, so a subdomain needs its own.
Primary sources
- OpenAI — bots and crawlersThe three OpenAI agents and what each is for.
- Perplexity — botsUser agents and published IP ranges.
- Anthropic — crawler accessClaudeBot and how to control it.
- Google — crawlers and fetchersWhere Google-Extended is defined.
- RFC 9309 — the robots.txt standardWhat crawlers are actually obliged to honour.
See how AI engines describe you right now.
The free scan checks the technical signals in this article against your homepage — schema, rendering, crawler access, entity clarity — and scores each one.
Analyze My Website