Claude: Make Your Documentation Discoverable by AI
AI Can't Find Your Documentation on Claude? Here's why it happens, the fix, and how to prevent it.
Claude Context
Make your website visible and citable by Claude, Anthropic's AI assistant known for nuanced understanding and detailed responses.
Allow Claude-SearchBot and Claude-User in robots.txt so Claude can find and read your pages
Focus on demonstrating genuine expertise (E-E-A-T signals)
Provide detailed, well-researched content with citations
Why This Problem Matters
Users ask AI for help with your product but it can't find or cite your documentation. Get your docs into AI knowledge.
Why This Happens in Claude
Documentation is only usable if Claude can reach it through Claude's web search, which Claude-SearchBot crawls for. Docs behind a login, on a blocked subdomain, or rendered client-side are invisible however good they are.
Signs You Have This Problem in Claude
Claude answers questions about your product without citing your docs
Claude gives generic answers instead of citing your docs
Users report AI couldn't help them with your product
Technical questions get wrong or incomplete answers
Competitor documentation gets cited instead of yours
How to Fix This in Claude
Follow these Claude-specific solutions:
Allow Claude-SearchBot in robots.txt, on your docs subdomain too
Serve docs as server-rendered HTML with one task per page
Make documentation publicly accessible without login
Publish docs as HTML pages, not just PDFs or apps
Implement proper heading hierarchy in docs
Link documentation prominently from your main site
Add TechArticle and HowTo schema to doc pages
Claude Technical Details
Anthropic documents three agents that all honour robots.txt: Claude-SearchBot for search results, ClaudeBot for training, and Claude-User for pages a user asks Claude to read.
Claude-SearchBot
Search / answers
Improves the quality of Claude's search results.
ClaudeBot
Model training
Collects public web content for model training.
Claude-User
User-requested fetch
Retrieves a page when a Claude user asks a question that needs it.
robots.txt rule to stay eligible for Claude answers:
User-agent: Claude-SearchBot
Allow: /Source: support.claude.com
Prevent This Problem in Claude
Design documentation for both humans and AI
Test AI's ability to answer questions from your docs
Maintain clear documentation structure
Monitor documentation crawling in server logs
Helpful Resources
How-To Guides
Checklists
Need Help Fixing Claude Issues?
Get expert help resolving this and other Claude visibility problems.
Get Your AI Visibility AuditFrequently Asked Questions
Why is this happening in Claude?
Documentation is only usable if Claude can reach it through Claude's web search, which Claude-SearchBot crawls for. Docs behind a login, on a blocked subdomain, or rendered client-side are invisible however good they are.
What is the first fix to prioritize in Claude?
Allow Claude-SearchBot in robots.txt, on your docs subdomain too