Perplexity: Unblock AI Crawlers from Your Website
AI Crawlers Blocked by Robots.txt on Perplexity? Here's why it happens, the fix, and how to prevent it.
Perplexity Context
Ensure your website appears in Perplexity's AI-powered search results with direct citations and source links.
Allow PerplexityBot in your robots.txt for indexing
Create content that directly answers specific questions
Include statistics, data points, and quotable information
Why This Problem Matters
Your robots.txt is preventing Perplexity from accessing your content. This simple fix can dramatically improve your AI visibility.
Why This Happens in Perplexity
PerplexityBot indexes pages for Perplexity's answers and is not used to train foundation models; Perplexity-User fetches pages on a user's request and generally ignores robots.txt. If robots.txt, a firewall or bot protection stops PerplexityBot and Perplexity-User, Perplexity can't use your pages however good they are.
Signs You Have This Problem in Perplexity
PerplexityBot is blocked from crawling your site, so Perplexity can't index it
Perplexity's real-time search can't access your pages
Server logs show no PerplexityBot visits
Perplexity can't cite you because it can't reach your content
How to Fix This in Perplexity
Follow these Perplexity-specific solutions:
Add to robots.txt: User-agent: PerplexityBot\nAllow: /
Ensure real-time crawling isn't blocked
Remove restrictions affecting PerplexityBot
Monitor server logs for PerplexityBot activity
Test Perplexity search to verify access
Keep robots.txt rules permissive for AI crawlers
Perplexity Technical Details
PerplexityBot indexes pages for Perplexity's answers and is not used to train foundation models; Perplexity-User fetches pages on a user's request and generally ignores robots.txt.
PerplexityBot
Search / answers
Surfaces and links sites in Perplexity answers; not used for foundation-model training.
Perplexity-User
User-requested fetch
Fetches a page when a user asks; generally ignores robots.txt.
robots.txt rule to stay eligible for Perplexity answers:
User-agent: PerplexityBot
Allow: /Source: docs.perplexity.ai
Prevent This Problem in Perplexity
Audit robots.txt (for PerplexityBot) quarterly
Stay updated on new AI crawler user agents
Test AI crawler access after any robots.txt (for PerplexityBot) changes
Balance security needs with AI visibility goals
Helpful Resources
Perplexity Guide
Complete Perplexity optimization guide
How-To Guides
Checklists
Need Help Fixing Perplexity Issues?
Get expert help resolving this and other Perplexity visibility problems.
Get Your AI Visibility AuditFrequently Asked Questions
Why is this happening in Perplexity?
PerplexityBot indexes pages for Perplexity's answers and is not used to train foundation models; Perplexity-User fetches pages on a user's request and generally ignores robots.txt. If robots.txt, a firewall or bot protection stops PerplexityBot and Perplexity-User, Perplexity can't use your pages however good they are.
What is the first fix to prioritize in Perplexity?
Add to robots.txt: User-agent: PerplexityBot\nAllow: /