robots.txt for AI crawlers: which bots to allow
The distinction that matters
Most sites that block AI crawlers block all of them with a single sweeping rule, having intended only to opt out of model training. Those are two different things.
Training crawlers build corpora used to train future models. Opting out is a legitimate editorial and commercial decision, and it does not affect whether an assistant can cite you today.
Retrieval crawlers fetch a page right now because a user asked a question that your page might answer. Blocking these removes you from the pool of citable sources — including for people searching your own brand name.
Who is who
| Crawler | Vendor | Role |
|---|---|---|
| OAI-SearchBot | OpenAI | Retrieval — builds the ChatGPT search index |
| ChatGPT-User | OpenAI | Retrieval — fetches a page during a conversation |
| GPTBot | OpenAI | Training |
| Claude-SearchBot | Anthropic | Retrieval — search index |
| Claude-User | Anthropic | Retrieval — live fetch |
| ClaudeBot | Anthropic | Training |
| PerplexityBot | Perplexity | Retrieval — search index |
| Perplexity-User | Perplexity | Retrieval — live fetch |
| Google-Extended | Training (Gemini). Does not affect Google Search | |
| Applebot-Extended | Apple | Training |
| CCBot | Common Crawl | Training corpus used by many labs |
A configuration that opts out of training but stays citable
# Retrieval crawlers — keep these allowed to remain citable User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Training crawlers — blocked in this example User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://yoursite.com/sitemap.xml
robots.txt is not the only gate
A permissive robots.txt does not mean crawlers can actually reach you. Bot-management rules at your CDN operate at a different layer and will return 403 regardless of what robots.txt says.
This is the single most common false negative in AI-readability audits: tools that read robots.txt report you as open, while the actual request is refused. Test with real requests, not just configuration.
Frequently asked
Does blocking Google-Extended affect my Google Search rankings?
No. Google-Extended controls use of your content for Gemini training. It is separate from Googlebot and does not affect Search ranking or indexing.
Do AI crawlers obey robots.txt?
The major crawlers from OpenAI, Anthropic, Perplexity and Google publish their user-agent tokens and state that they respect robots.txt. Enforcement against crawlers that ignore it requires blocking at the network layer.
Is a missing robots.txt a problem?
Not for access — no robots.txt means crawling is allowed by default. You do lose the ability to declare a sitemap and to state per-crawler preferences.
Last updated 2026-07-26.
Check your own site
Runs every check in this guide against fourteen AI crawlers. Free, no account.
No signup. No credit card. Results are a shareable link.