AI Crawler
An AI crawler is an automated agent operated by an AI company that fetches web pages either to train models or to support live search within an assistant.
The distinction between the two purposes is the part most sites get wrong. Training crawlers collect data for future models; search crawlers fetch pages to answer questions now. They are usually separate user agents with separate robots.txt rules.
Blocking a training crawler is a legitimate content licensing decision. Blocking a search crawler removes you from that engine's answers, which is almost never what anyone intends.
Why it matters
Sites regularly block the wrong agent and quietly remove themselves from AI search while believing they protected their content from training.
An example
A publisher adds a rule to robots.txt blocking every user agent containing "GPT" or "Claude", intending to keep its articles out of model training. The rule also blocks the search agents those companies use to answer live questions, and within weeks the publisher stops appearing in those assistants' answers entirely.
Common mistakes
- Blocking by vendor name rather than by agent. Each vendor runs separate training and search crawlers.
- Trusting robots.txt alone. A CDN or firewall rule can block a crawler that robots.txt allows.
Questions
- How do I check which AI crawlers can access my site?
- Read robots.txt as each crawler would, then fetch a page using each crawler's user agent to confirm nothing further along — a firewall or bot protection service — is refusing it. Both checks are needed, because either can block access independently.
- Should I block AI crawlers?
- Block training crawlers if you do not want your content used to train models; that is a legitimate choice. Keep search crawlers allowed if you want to appear in AI answers. The two decisions are independent.