AI crawler user-agent reference

The reference table

This table covers the documented crawlers and controls used by the major answer engines as of 29 July 2026. “Not documented” means the vendor has not made a public promise about JavaScript execution. It does not mean that a browser render was observed.

Versions inside a user-agent can change. Match the stable token in robots.txt and logs. Verify source IPs where the vendor publishes them, since anyone can copy a user-agent string.

Exact published strings

These are the full examples the vendors publish. OpenAI, Amazon, Apple, Google and Microsoft say that version fields can change. A firewall rule should match the stable bot token and, when available, the vendor IP list. A user-agent match alone does not prove who sent the request.

# OpenAI search
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

# OpenAI user fetch
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

# OpenAI training
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

# Perplexity search
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

# Perplexity user fetch
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

# Amazon search. Replace W.X.Y.Z with a changing Chrome version when matching logs.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36

# Amazon user fetch
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36

# Amazon training and product improvement
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

# Apple search, desktop example
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

# Common Crawl
CCBot/2.0 (https://commoncrawl.org/faq/)

Two names that never appear in HTTP logs

Google-Extended is a robots.txt product token. Google says it has no separate HTTP user-agent. Googlebot performs the crawl, while Google-Extended controls whether the resulting content may be used for future Gemini training and grounding in Gemini products. Blocking it does not remove a page from Google Search.

Applebot-Extended works the same way. Apple says it never crawls pages. Applebot performs the request, while Applebot-Extended controls whether Apple may use that content to train foundation models. Blocking Applebot-Extended does not block Apple search.

Rules that look for either Extended token in server logs will never match legitimate crawler traffic. Put those tokens in robots.txt only.

# Keep Google and Apple search crawling, opt out of model training
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Search, live fetch and training are separate choices

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Amzn-SearchBot build indexes used in answers. Blocking one can remove your pages from that vendor’s search results.

ChatGPT-User, Claude-User, Perplexity-User and Amzn-User fetch a page because a person asked for it. OpenAI and Perplexity say robots.txt may not apply to these user-triggered requests. A WAF can still refuse the HTTP request.

GPTBot and ClaudeBot are training crawlers. Amazonbot may be used for training. CCBot builds a public corpus that many organizations can use, including for training. Blocking a training crawler is not the same decision as blocking a search crawler.

JavaScript support is the exception, not a safe default

Apple explicitly says Applebot may render a page in a browser. Google documents Googlebot as a rendering crawler. Microsoft recommends dynamic rendering for Bingbot when client-side JavaScript prevents indexing. Common Crawl explicitly says CCBot does not execute JavaScript.

OpenAI, Anthropic, Perplexity, Amazon and Meta do not publish a JavaScript-rendering promise for the agents above. Serve the answer-bearing content in the initial HTML. A crawler can advertise Chrome in its user-agent without running a browser.

Check a real request without guessing

Send the vendor’s published user-agent, record the final status, then inspect the raw response for text that matters. This tests the path through your CDN and origin. It does not authenticate the vendor and it does not prove what a future crawl will do.

UA='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'

curl -sL -A "$UA" -o /tmp/oai-page.html -w 'HTTP %{http_code}\n' https://yoursite.com/
grep -i 'a sentence only your real page contains' /tmp/oai-page.html

# Also inspect the policy the real crawler will read
curl -s https://yoursite.com/robots.txt

Primary sources

Vendor pages are the source of truth. The table records what each vendor documents, including when a full HTTP string or rendering behavior is not published.

• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Amazon crawler documentation
• Applebot documentation
• Google crawler documentation
• Microsoft Bingbot documentation
• Common Crawl CCBot documentation
• Meta crawler documentation

Frequently asked