Claude: Unblock AI Crawlers from Your Website
AI Crawlers Blocked by Robots.txt on Claude? Here's why it happens, the fix, and how to prevent it.
Claude Context
Make your website visible and citable by Claude, Anthropic's AI assistant known for nuanced understanding and detailed responses.
Allow Claude-SearchBot and Claude-User in robots.txt so Claude can find and read your pages
Focus on demonstrating genuine expertise (E-E-A-T signals)
Provide detailed, well-researched content with citations
Why This Problem Matters
Your robots.txt is preventing Claude from accessing your content. This simple fix can dramatically improve your AI visibility.
Why This Happens in Claude
Anthropic documents three agents that all honour robots.txt: Claude-SearchBot for search results, ClaudeBot for training, and Claude-User for pages a user asks Claude to read. If robots.txt, a firewall or bot protection stops Claude-SearchBot and Claude-User, Claude can't use your pages however good they are.
Signs You Have This Problem in Claude
Claude-SearchBot or Claude-User is blocked in robots.txt
Claude's crawler can't index your content
Server logs show zero visits from Claude-SearchBot
Claude can't access your site to build knowledge
How to Fix This in Claude
Follow these Claude-specific solutions:
Add to robots.txt: User-agent: Claude-SearchBot\nAllow: /
Also allow: User-agent: Claude-User\nAllow: /
Remove overly restrictive Disallow rules
Monitor server logs for ClaudeBot visits
Test that Claude can access key pages
Create an llms.txt file with AI-specific guidance
Claude Technical Details
Anthropic documents three agents that all honour robots.txt: Claude-SearchBot for search results, ClaudeBot for training, and Claude-User for pages a user asks Claude to read.
Claude-SearchBot
Search / answers
Improves the quality of Claude's search results.
ClaudeBot
Model training
Collects public web content for model training.
Claude-User
User-requested fetch
Retrieves a page when a Claude user asks a question that needs it.
robots.txt rule to stay eligible for Claude answers:
User-agent: Claude-SearchBot
Allow: /Source: support.claude.com
Prevent This Problem in Claude
Audit robots.txt (for Claude-SearchBot) quarterly
Stay updated on new AI crawler user agents
Test AI crawler access after any robots.txt (for Claude-SearchBot) changes
Balance security needs with AI visibility goals
Helpful Resources
How-To Guides
Checklists
Need Help Fixing Claude Issues?
Get expert help resolving this and other Claude visibility problems.
Get Your AI Visibility AuditFrequently Asked Questions
Why is this happening in Claude?
Anthropic documents three agents that all honour robots.txt: Claude-SearchBot for search results, ClaudeBot for training, and Claude-User for pages a user asks Claude to read. If robots.txt, a firewall or bot protection stops Claude-SearchBot and Claude-User, Claude can't use your pages however good they are.
What is the first fix to prioritize in Claude?
Add to robots.txt: User-agent: Claude-SearchBot\nAllow: /