Configure Robots.txt for AI Visibility

Allow OAI-SearchBot (ChatGPT search)

User-agent: OAI-SearchBot with Allow: /. Sites that block it are not shown in ChatGPT search answers

Allow ChatGPT-User

Fetches a page when a user asks ChatGPT to read it

Allow Claude-SearchBot and Claude-User

Anthropic's search crawler and user-requested fetcher

Allow PerplexityBot

Perplexity's search crawler; not used for model training

Allow Meta-WebIndexer and MistralAI-Index

Meta AI's and Mistral's search-index crawlers

Keep the Google-Extended token allowed

A token, not a crawler: controls Gemini training and grounding, not Google Search

Decide on training crawlers separately

GPTBot, ClaudeBot, Meta-ExternalAgent and MistralAI-Training only collect training data; blocking them doesn't remove you from AI search

Don't add rules for bots vendors don't document

You.com, xAI and Cohere publish no crawler; Brave's crawler follows your Googlebot rules

Repeat Disallow rules in AI groups

Order doesn't matter: a crawler that matches a named group ignores the * group, so copy your private-path disallows into it

Don't use overly aggressive crawl-delay

May prevent full indexing

Allow access to CSS/JS files

AI needs these to render pages

Include sitemap directive

Help crawlers discover pages

Block admin areas

Disallow: /admin/

Block login/auth pages

No value for AI indexing

Block duplicate content paths

Prevent confusion

Block internal search results

Low-value pages

Test with robots.txt validator

Use Google's testing tool

Check server logs for AI bots

Verify crawlers are accessing

Monitor for new AI crawlers

Stay updated on new bots

Review quarterly

Update as AI landscape changes