How to allow AI crawlers through Cloudflare

Allow retrieval without opening training

Cloudflare now separates AI traffic by purpose. AI Search bots build indexes used in answers. AI Assistant bots fetch a page because a person asked for it. AI Crawler is Cloudflare’s category for training and other collection bots.

For visibility in live answers, allow AI Search and AI Assistant traffic. Training remains a separate policy. This avoids the old all-or-nothing choice behind the single Block AI bots toggle.

Step 1: allow each retrieval crawler

In the Cloudflare dashboard, select the account and domain. Open AI Crawl Control, then Crawlers. Search for each crawler below and choose Allow in the Action column.

Set OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User to Allow. Add Applebot if Siri, Spotlight and Apple’s AI search features matter to you.

The Allow action means AI Crawl Control does not add a blocking rule for that crawler. It does not override a different WAF rule, bot policy or rate limit. Cloudflare’s own troubleshooting guide says an allowed crawler can still be stopped by an earlier security rule.

Step 2: allow the Search and Agent behaviors

Open Security Settings and select Configure AI bot policies. Set Search to Allow (do not block). Set Agent to Allow (do not block). Choose the Training setting separately.

Cloudflare introduced these behavior controls on 1 July 2026. On 15 September 2026, its announced defaults for new domains will keep Search allowed while blocking Agent and Training on pages that display ads. A new zone can therefore allow search indexing but still refuse a live ChatGPT-User or Claude-User fetch unless Agent is changed to Allow.

If your dashboard still shows the legacy Block AI bots control, remember that it is being retired on 15 September 2026. Review the three behavior settings instead of treating the legacy toggle as the final state.

Step 3: find the rule that still blocks the request

Open Security > WAF > Custom rules. Look for the AI Crawl Control rule and any earlier rule that blocks, challenges or rate limits automated traffic. Custom rules run before Cloudflare bot settings. A terminating action in an earlier rule prevents the request from reaching the allow settings you just changed.

Use Security Events to find the exact product and rule that acted on the request. Change only that rule. A country block, login rate limit or managed-rule exception may be intentional on other paths.

On paid plans, a Skip rule can bypass selected WAF products and Super Bot Fight Mode for verified traffic. Cloudflare does not let a Skip rule bypass Bot Fight Mode on the Free plan. If Security Events names Bot Fight Mode there, turn Bot Fight Mode off and keep the narrower AI policies enabled.

Enterprise: skip only the verified crawlers you chose

Enterprise Bot Management customers can match Cloudflare detection IDs instead of trusting a user-agent string. Create a custom rule above the rule causing the block. Use the expression below, choose Skip, and select only the security product that the event log identified.

The IDs come from Cloudflare’s bot reference as published on 30 July 2026. Check the current reference before copying this rule into a long-lived configuration. Cloudflare can add detection IDs as its bot identification changes.

any(cf.bot_management.detection_ids[*] in {
  126255384 33563986
  132995013 33563857
  33564301
  33564303
  33563889
  33564371
})

Step 4: make robots.txt agree

A network allow and a robots.txt allow solve different gates. The request must pass Cloudflare first, then the crawler reads robots.txt. Keep the search and user-fetch tokens open if you want retrieval. Put any training opt-out under the training crawler tokens.

Cloudflare managed robots.txt can add training directives. Check AI Crawl Control > Directives to see whether it is enabled and what the live file says.

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Training is a separate choice
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Step 5: verify status and content

Run both requests and compare the final status, response size and a sentence that only the real page contains. A 200 with a challenge page is still a failure, so checking the status alone is not enough.

This curl test changes only the claimed user-agent. It catches user-agent rules and Cloudflare’s free-plan AI Crawl Control detection. It does not prove that a request came from a vendor IP. For identity-based Enterprise rules, confirm an actual verified request in Security Events or AI Crawl Control metrics.

ORIGIN='https://yoursite.com'
URL="$ORIGIN/"
NEEDLE='a sentence from your real page'
UA='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot'

# Ordinary request
curl -sL -o /tmp/browser.html -w 'browser: %{http_code} %{size_download} bytes\n' "$URL"

# Same URL with the crawler token
curl -sL -A "$UA" -o /tmp/oai.html -w 'crawler: %{http_code} %{size_download} bytes\n' "$URL"

# The real page text must be present in the crawler response
grep -Fi "$NEEDLE" /tmp/oai.html

# Then inspect the policy file
curl -sL "$ORIGIN/robots.txt"

What each failed check means

Primary sources

Cloudflare’s documentation is the source for the dashboard paths, execution order, plan limits, behavior controls and detection IDs in this walkthrough.

• Manage AI crawlers in AI Crawl Control
• Configure AI bot policies
• AI Crawl Control and WAF execution order
• Configure a WAF Skip rule
• Cloudflare AI crawler detection IDs
• Track managed robots.txt directives

Frequently asked