Glossary
AI Crawler
An AI crawler is an automated agent operated by an AI company that fetches web pages either to train models or to support live search within an assistant.
The distinction between the two purposes is the part most sites get wrong. Training crawlers collect data for future models; search crawlers fetch pages to answer questions now. They are usually separate user agents with separate robots.txt rules.
Blocking a training crawler is a legitimate content licensing decision. Blocking a search crawler removes you from that engine's answers, which is almost never what anyone intends.
Why it matters
Sites regularly block the wrong agent and quietly remove themselves from AI search while believing they protected their content from training.