Guides
How AI crawlers actually work
Reproducible checks and honest assessments. Where the evidence is thin, we say so.
Do AI crawlers render JavaScript?
None of the major AI crawlers execute JavaScript. Here is what that means for client-side rendered sites, and how to check yours.
robots.txt for AI crawlers: which bots to allow
AI crawlers are not interchangeable. Blocking the retrieval bots removes you from live AI answers; blocking training bots does not.
Is your CDN blocking AI crawlers without telling you?
Bot-management rules reject AI crawlers at the edge regardless of robots.txt. Here is how to find and fix it.
What is llms.txt, and does it do anything?
An honest assessment of llms.txt: what it proposes, what evidence exists that anything consumes it, and where it belongs on your list.
How to check whether ChatGPT can see your website
A reproducible three-step check for whether ChatGPT, Claude and Perplexity can reach and read your site.