Robots.txt Generator

Build a robots.txt file with an interactive editor. Configure rules for search engines and AI crawlers with presets and syntax validation.

Quick Add Bot Rules

User-agent

Common paths:

Sitemap URL (optional)

Host (optional, for Yandex)

User-agent: *
Disallow: /admin/

GPTBot:OpenAI's crawler for training data. Block to prevent content use in AI training.
Google-Extended:Google's AI training crawler. Separate from regular Googlebot.
CCBot:Common Crawl's bot, used by many AI companies for training data.
Claude-Web/anthropic-ai:Anthropic's crawlers for Claude.
PerplexityBot:Perplexity AI's crawler for search results.

Note: Blocking AI crawlers prevents training but may reduce AI visibility. Consider your goals carefully.