getcourtyard.ai
Scanned Thu, 30 Jul 2026 04:56:03 GMT · https://getcourtyard.ai/research/how-ai-reads-a-knowledge-base
This site is readable by AI assistants
No blocking issues found. Remaining items are refinements.
Keep a copy of this report
We’ll email the full findings plus a prioritised fix checklist for getcourtyard.ai. Access failures come first, followed by response-content and structure fixes. No account needed.
Initial response and browser render
Word counts of readable text before and after browser execution. The gap is a rendering dependency, not a claim about every vendor’s crawler behavior.
Crawler access
Live requests were sent with each vendor’s published user-agent string. This tests how the site handles the claimed identity; a user-agent alone does not authenticate the vendor. A robots.txt pass with a failed request points to another access control.
| Crawler | Vendor | Role | robots.txt | Live request |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Allowed | HTTP 200 |
| OAI-SearchBot | OpenAI | Search indexretrieval | Allowed | HTTP 200 |
| ChatGPT-User | OpenAI | Live fetchretrieval | Allowed | not probed |
| ClaudeBot | Anthropic | Training | Allowed | HTTP 200 |
| Claude-SearchBot | Anthropic | Search indexretrieval | Allowed | not probed |
| Claude-User | Anthropic | Live fetchretrieval | Allowed | not probed |
| PerplexityBot | Perplexity | Search indexretrieval | Allowed | HTTP 200 |
| Perplexity-User | Perplexity | Live fetchretrieval | Allowed | not probed |
| Google-Extended | Training | Allowed | not probed | |
| Applebot-Extended | Apple | Training | Allowed | not probed |
| meta-externalagent | Meta | Training | Allowed | not probed |
| Amazonbot | Amazon | Search index | Allowed | not probed |
| Bytespider | ByteDance | Training | Allowed | not probed |
| CCBot | Common Crawl | Training | Allowed | not probed |
Findings (1)
No blocking issues. The items below are refinements.
No llms.txt
- Evidence
- https://getcourtyard.ai/llms.txt returns an HTML page rather than a plain-text file (soft 404).
- Why it matters
- An emerging, not-yet-universally-consumed convention: a plain-text map of your most important pages for language models. Low cost, speculative upside.
- Fix
- Publish /llms.txt listing your key pages with one-line descriptions.
Keep it that way
The diagnosis stays free. Subscribe when you want scheduled re-scans, history and an alert if a later check regresses.