Robots.txt Generator

Build a robots.txt file with an interactive editor. Configure rules for search engines and AI crawlers with presets and syntax validation.

Quick Add Bot Rules

User-agent

Common paths:

Sitemap URL (optional)

Host (optional, for Yandex)

User-agent: *
Disallow: /admin/

OAI-SearchBot: decides whether pages can appear in ChatGPT search answers. Blocking it removes you from ChatGPT search.
GPTBot: OpenAI's crawler for training data. Block it to opt out of training without leaving ChatGPT search.
Google-Extended: a robots.txt token, not a separate crawler (Googlebot does the fetching). Disallowing it opts content out of Gemini training and grounding without affecting Google Search.
CCBot: Common Crawl's bot, used by many AI companies for training data.
Claude-SearchBot / ClaudeBot / Claude-User: Anthropic's agents for Claude search, model training, and user-requested fetches.
PerplexityBot: Perplexity's crawler for search results; not used for model training.

Note: Blocking AI crawlers prevents training but may reduce AI visibility. Consider your goals carefully.