Perplexity: Unblock AI Crawlers from Your Website
AI Crawlers Blocked by Robots.txt on Perplexity? Here's the step-by-step fix — the causes, the exact solution, and how to prevent it long-term.
Perplexity Context
Ensure your website appears in Perplexity's AI-powered search results with direct citations and source links.
Allow PerplexityBot in your robots.txt for indexing
Create content that directly answers specific questions
Include statistics, data points, and quotable information
Why This Problem Matters
Your robots.txt is preventing AI systems from accessing your content. This simple fix can dramatically improve your AI visibility.
No AI-referred traffic despite good traditional SEO
AI systems have no information about your brand
Server logs show no visits from GPTBot, ClaudeBot, etc.
Why This Happens in Perplexity
Perplexity uses PerplexityBot to discover and index content. Blanket 'Disallow: /' rules affecting AI bots. To fix this, you need to ensure Perplexity's crawler can access your content and that your pages are structured in a way Perplexity can understand.
Signs You Have This Problem in Perplexity
PerplexityBot is blocked from crawling your site
Perplexity's real-time search can't access your pages
Server logs show no PerplexityBot visits
Perplexity can't cite you because it can't reach your content
How to Fix This in Perplexity
Follow these Perplexity-specific solutions:
Add to robots.txt: User-agent: PerplexityBot\nAllow: /
Ensure real-time crawling isn't blocked
Remove restrictions affecting PerplexityBot
Monitor server logs for PerplexityBot activity
Test Perplexity search to verify access
Keep robots.txt rules permissive for AI crawlers
Perplexity Technical Details
Crawler Name
PerplexityBot
User Agent
PerplexityBot
Required robots.txt configuration:
User-agent: PerplexityBot
Allow: /Prevent This Problem in Perplexity
Audit robots.txt (for PerplexityBot) quarterly
Stay updated on new Perplexity crawler user agents
Test Perplexity crawler access after any robots.txt (for PerplexityBot) changes
Balance security needs with Perplexity visibility goals
Helpful Resources
Perplexity Guide
Complete Perplexity optimization guide
How-To Guides
Checklists
Frequently Asked Questions
Why does ai crawlers blocked by robots.txt happen in Perplexity?
Perplexity uses PerplexityBot to discover and index content. Blanket 'Disallow: /' rules affecting AI bots. To fix this, you need to ensure Perplexity's crawler can access your content and that your pages are structured in a way Perplexity can understand.
What is the first fix to prioritize in Perplexity?
Add to robots.txt: User-agent: PerplexityBot\nAllow: /