Robots.txt allows AI crawlers. Why is Cloudflare still blocking them?

A permissive robots.txt is only the first gate. A CDN or WAF can still challenge or reject a crawler before it receives your page. If OAI-SearchBot is allowed in robots.txt but access logs show 403s, challenges or no successful fetches, inspect your edge security events and bot settings. Change the smallest rule that blocks a verified crawler; do not turn off protection for every bot.

This is a different diagnosis from choosing which AI crawlers to allow. That guide covers crawler policy. This one covers delivery. Cloudflare documents the interaction between AI Crawl Control and WAF rules: an allowed AI crawler can still be blocked by an earlier WAF rule.

Two gates to check, in order

Open the production robots file, not a local build or the www variant if the canonical host is bare. Confirm that a specific User-agent group and its path rules permit the page you want available. A broad Allow: / does not override a more specific applicable disallow under every possible configuration; use a robots tester or inspect the matching group. RankVyze's AI Crawler Checker can flag obvious policy problems, but it cannot prove that a real platform crawler passed your WAF.

Find the block in security events

1. 1

   Pick a page and a time window

   Choose one canonical URL, note its path, and inspect the CDN's security events for recent crawler requests to that host. Search for the relevant user agent, bot classification, rule ID, and request action.

2. 2

   Verify the caller

   Do not trust a user-agent string by itself; anybody can copy it. Prefer your CDN's verified-bot classification or the platform's published verification method before changing a firewall rule.

3. 3

   Read the action

   A block, managed challenge, JavaScript challenge or rate limit can stop a non-browser crawler. Check whether the event was caused by a custom rule, bot setting, AI Crawl Control, an origin firewall or a host-level access policy.

4. 4

   Narrow the exception

   If the request is verified and you intend to allow it, scope the exception to that bot/category and necessary public paths. Keep private account, checkout and admin routes protected.

5. 5

   Retest with real evidence

   Look for a later verified request that receives the intended page with a successful status. Save the event ID, rule changed and a sample response. Then monitor rather than assuming immediate inclusion in an AI answer.

Cloudflare's bot reference distinguishes OAI-SearchBot, GPTBot, ChatGPT-User, Claude-SearchBot and PerplexityBot. These names do not all serve the same purpose. Cloudflare also documents a known-bot field for custom rules. The exact controls available depend on your plan and configuration, so start from the event that blocked the request instead of copying a broad rule from a blog post.

Why a curl test is not enough

You can run curl -I -A 'OAI-SearchBot' https://example.com/page to check what your server does with that string. It is a useful smoke test for redirects and accidental blocks. It is not proof that OpenAI can crawl you: the request comes from your machine, with a spoofed user agent and a different IP, bot score and challenge context. The reverse is also true: a 403 to your spoofed request does not prove the real verified crawler is blocked. CDN events for verified requests are stronger evidence.

If the event log shows no crawler requests

Do not create an allow rule to solve a request you have not observed. Check whether the page is linked from discoverable pages, is publicly available without a login, has a stable canonical URL and actually returns useful text. Compare the relevant search engine's indexing status where one exists. Then record the absence as an observation, not proof that the crawler has blacklisted the site. Review our indexing guide and AI rank checker for separate discovery and answer checks.