Is your CDN blocking AI crawlers without telling you?

Why this is missed so often

Audit tools that check "can AI crawlers access this site" almost always read robots.txt and stop there. robots.txt is a text file stating a policy. It has no power to make a request succeed.

If your CDN is configured to challenge or block non-browser traffic, the crawler gets a 403 or an interstitial challenge page it cannot solve. From the crawler's perspective your site has no content. From your dashboard, everything looks fine.

Common causes

Cloudflare Bot Fight Mode challenges automated traffic broadly and catches well-behaved crawlers alongside scrapers. Cloudflare also offers an explicit "Block AI Scrapers and Crawlers" toggle, which some accounts have enabled by default. Similar features exist across other providers under names like bot management, bot protection or advanced rate limiting.

Aggressive rate limiting produces the same outcome intermittently — a crawler that gets 429 responses will back off and may not return.

How to test properly

Send a real request with a crawler user-agent and compare it against a browser request. A different status code, or a much smaller response, means the edge is interfering.

# As a browser
curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com

# As a retrieval crawler
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" \
  https://yoursite.com

Fixing it

Allowlist the verified AI crawlers you want in your bot-management rules. Verify them by reverse DNS or published IP ranges rather than by user-agent string alone, since the string itself is trivially spoofed.

Then re-test with real requests. Configuration changes at the edge do not always take effect where you expect, and the only reliable confirmation is a request that returns 200 with your content in it.

Frequently asked