blog.dogtorcito.com
Scanned Fri, 07 Aug 2026 11:09:02 GMT · https://blog.dogtorcito.com/
Critical access or response-content failures found
Your server or CDN blocks 4 AI crawlers at the network layer
Send this to whoever can deploy the fix
We’ll email the full findings plus a prioritised fix checklist for blog.dogtorcito.com. Access failures come first, followed by response-content and structure fixes. No account needed.
Initial response and browser render
Word counts of readable text before and after browser execution. The gap is a rendering dependency, not a claim about every vendor’s crawler behavior.
Crawler access
Live requests were sent with each vendor’s published user-agent string. This tests how the site handles the claimed identity; a user-agent alone does not authenticate the vendor. A robots.txt pass with a failed request points to another access control.
| Crawler | Vendor | Role | robots.txt | Live request |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | Blocked | HTTP 403 |
| OAI-SearchBot | OpenAI | Search indexretrieval | Allowed | HTTP 403 |
| ChatGPT-User | OpenAI | Live fetchretrieval | Allowed | not probed |
| ClaudeBot | Anthropic | Training | Blocked | HTTP 403 |
| Claude-SearchBot | Anthropic | Search indexretrieval | Allowed | not probed |
| Claude-User | Anthropic | Live fetchretrieval | Allowed | not probed |
| PerplexityBot | Perplexity | Search indexretrieval | Allowed | HTTP 403 |
| Perplexity-User | Perplexity | Live fetchretrieval | Allowed | not probed |
| Google-Extended | Training | Blocked | not probed | |
| Applebot-Extended | Apple | Training | Blocked | not probed |
| meta-externalagent | Meta | Training | Blocked | not probed |
| Amazonbot | Amazon | Search index | Blocked | not probed |
| Bytespider | ByteDance | Training | Blocked | not probed |
| CCBot | Common Crawl | Training | Blocked | not probed |
Findings (3)
1 critical access or response issue found in this scan.
Your server or CDN blocks 4 AI crawlers at the network layer
- Evidence
- OAI-SearchBot: HTTP 403; PerplexityBot: HTTP 403; ClaudeBot: HTTP 403; GPTBot: HTTP 403
- Why it matters
- robots.txt permits OAI-SearchBot, PerplexityBot, but the live test using those published identities was refused before it reached the content. A robots.txt-only check would miss this access layer.
- Fix
- Review the WAF or bot-management event that handled the request. Allow the crawler identities you chose. For a security bypass, verify vendor IP ranges or reverse DNS instead of trusting a user-agent string alone.
No question-shaped content
- Evidence
- No question headings and no FAQPage schema found.
- Why it matters
- Assistants answer questions. Pages organised as explicit question/answer pairs are markedly easier to retrieve and quote than undifferentiated marketing prose.
- Fix
- Add a server-rendered FAQ section using the real questions prospects ask, marked up with FAQPage schema.
robots.txt blocks 8 AI training crawlers
- Evidence
- GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), meta-externalagent (Meta), Amazonbot (Amazon), Bytespider (ByteDance), CCBot (Common Crawl)
- Why it matters
- This can be a deliberate publisher choice. These controls apply to training use, while vendors document separate agents for search and user-triggered retrieval.
- Fix
- No action needed if this is intentional. Verify retrieval crawlers remain allowed.
Get these fixed and keep them fixed
The diagnosis stays free. Paid plans generate reviewable robots.txt, JSON-LD and llms.txt artifacts, then re-scan on schedule and email you when a check regresses.