GPTBot
GPTBot is OpenAI's web crawler used to collect training data; it is distinct from the agents that power search results shown inside ChatGPT.
OpenAI operates several agents with different purposes. GPTBot gathers training data. OAI-SearchBot supports search and the links shown in answers. ChatGPT-User fetches a page when a user follows a link.
Blocking GPTBot in robots.txt opts out of training use. It does not remove a site from ChatGPT's search results, because those use a different agent.
Why it matters
The confusion runs both ways: some sites block GPTBot expecting it to stop ChatGPT recommending competitors, and others allow everything while intending to opt out of training. Both are avoidable.
An example
A robots.txt file contains "User-agent: GPTBot" followed by "Disallow: /". The site's content will not be collected for OpenAI's model training. Its pages can still be fetched by OAI-SearchBot and shown in ChatGPT search results, because that agent is not covered by the rule.
Common mistakes
- Expecting a GPTBot block to remove a site from ChatGPT search.
- Blocking OAI-SearchBot by accident while intending to block only training.
Questions
- What is the difference between GPTBot and OAI-SearchBot?
- GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot fetches pages to power search results inside ChatGPT. They are separate user agents with separate robots.txt rules, so each can be allowed or blocked independently.
- How do I block GPTBot?
- Add a group to robots.txt with "User-agent: GPTBot" and "Disallow: /". That opts the site out of training collection without affecting ChatGPT search, which uses OAI-SearchBot.