All free tools
Can AI crawlers actually reach your site?
Check whether GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and Google-Extended can reach your site — in robots.txt and at the edge. Free, no signup.
- All 12 AI crawlers, including the control tokens that aren't crawlers
- Proper robots.txt parsing — wildcard groups, longest-match, Allow over Disallow
- A live request as each crawler, which catches CDN blocking robots.txt can't show
Free, no signup, no rate limit. Nothing you check is stored.
Questions about this tool
- How do I know if GPTBot is blocked?
- Enter your domain above. The tool reads your robots.txt, works out which rule applies to GPTBot — including a wildcard group that names no crawler — and then makes a real request identifying as GPTBot to see what your server actually returns.
- Why does this check the site as well as robots.txt?
- Because robots.txt is only a request. Cloudflare, other CDNs and WAFs can block AI crawlers at the edge regardless of what your file permits, and some enable that by default. A permissive robots.txt plus an edge block is the most common invisible failure in AEO.
- Is Google-Extended a crawler?
- No. It has no user agent and fetches nothing. Googlebot does the crawling; Google-Extended is a robots.txt token governing whether your content can be used for Gemini and grounded answers. Google states that disallowing it does not affect Search ranking.
- Should I allow all AI crawlers?
- For most businesses, yes — being known is the point. Publishers whose archive is the product sometimes block the training crawlers while allowing the search crawlers, which is exactly why the two are separate.
Ready to become the answer?
Free scan in ten seconds. $99 to fix it — refunded in full if we don't.