RankVyze logoRankVyze
Launch a product
All free tools

What do your AI crawler rules allow?

Check robots.txt rules for major AI agents and run user-agent probes to flag possible CDN or WAF blocks. Verify actual platform bot access in your server logs. Free, no signup.

  • 12 AI-related agents and control tokens, clearly distinguished
  • Proper robots.txt parsing — wildcard groups, longest-match, Allow over Disallow
  • A user-agent probe that can flag possible access problems for follow-up in CDN logs

Robots.txt permits the bot but the probe was refused? Diagnose CDN and WAF blocks using verified bot events.

Free, no signup, no rate limit. Nothing you check is stored.

How to use the ai crawler checker

  1. Enter your domain

    It reads your robots.txt and evaluates it the way each crawler does, rather than just showing you the file.

  2. Look at search crawlers first

    OAI-SearchBot, PerplexityBot and Claude-SearchBot decide whether you can appear in AI answers at all. Blocking these removes you completely.

  3. Decide about training crawlers separately

    GPTBot, ClaudeBot and Google-Extended feed model training. Blocking them is a legitimate choice and does not remove you from AI search.

  4. Check the live probe results

    robots.txt is only half the story. A CDN or firewall can block a crawler by user agent regardless of what your robots.txt permits.

The distinction that costs people the most visibility

Each AI vendor runs more than one crawler, and they do different jobs. OpenAI's GPTBot gathers training data; OAI-SearchBot retrieves pages to answer a live question. They are separate user agents and they obey separate rules. Sites that want to stay out of model training frequently block everything with the vendor's name in it, which also removes them from the answers themselves. That is close to the worst outcome available: no training benefit to anyone, and no visibility either.

Why robots.txt alone is not proof of access

robots.txt is a request that well-behaved crawlers honour, but it is not what enforces access. A bot-management rule at your CDN, a firewall filtering unknown user agents, or a rate limit can all refuse an AI crawler while your robots.txt says it is welcome. This is common on sites behind aggressive bot protection and it is invisible in the file itself, which is why this tool also fetches your homepage as each crawler and reports what actually came back.

Questions about this tool

How do I know if GPTBot is blocked?
Enter your domain above. The tool reads your robots.txt, works out which rule applies to GPTBot — including a wildcard group that names no crawler — and then sends a RankVyze probe using a GPTBot user-agent string. This is not a verified GPTBot request.
Why does this check the site as well as robots.txt?
Because robots.txt is only a request. Cloudflare, other CDNs and WAFs can block AI crawlers at the edge regardless of what your file permits, and some enable that by default. If our probe is refused, inspect verified bot events before concluding that the real crawler is blocked.
Is Google-Extended a crawler?
No. It has no user agent and fetches nothing. Googlebot does the crawling; Google-Extended is a robots.txt token governing whether your content can be used for Gemini and grounded answers. Google states that disallowing it does not affect Search ranking.
Should I allow all AI crawlers?
Decide by purpose. Search crawlers support retrieval, while training crawlers serve a different use. You can allow one group and disallow another; neither setting guarantees a citation or recommendation.

Other free tools

Meta Tag CheckerTitle, description, canonical, headings, Open Graph — with lengths, a SERP preview, and the no-JavaScript word count.AEO ScannerAll ten signals at once, scored out of 100.Run a scan

Put this resource into practice

YOUR NEXT MOVE

Launch today.
Keep growing tomorrow.

Give your product a home, then explore the tools that help you get discovered.

Launch your product for free Explore free SEO tools