All Templates
text template

Robots.txt Template for AI Crawler Access

A ready-to-use robots.txt configuration that allows the vendor-documented AI crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and more) while keeping private paths blocked for them too.

Template

# Robots.txt - AI Crawler Optimized
# Generated by Teoraspace
#
# How groups work: a crawler obeys only the most specific group that names it
# and ignores the * group. So every group below repeats the same Disallow lines;
# otherwise AI crawlers would be allowed into your private paths.

# ===================
# All other crawlers
# ===================
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /login/
Disallow: /account/

# ===================
# AI crawlers - ALLOW (vendor-documented tokens only)
# ===================
# Search / answer crawlers: these decide whether you can appear in AI answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: MistralAI-Index
# User-requested fetchers: read a page when a user asks the assistant to
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
# Microsoft Copilot answers from the Bing index
User-agent: bingbot
# Training crawlers: remove these lines to opt out of training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
# Google-Extended is a token, not a crawler: controls Gemini training/grounding
User-agent: Google-Extended
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Disallow: /login/
Disallow: /account/

# No rules needed for You.com, xAI/Grok or Cohere (no documented crawler).
# Brave's crawler follows your Googlebot rules.

# ===================
# Sitemap
# ===================
Sitemap: https://www.yoursite.com/sitemap.xml

How to Use This Template

1

Replace 'yoursite.com' with your actual domain

2

Add any additional private paths to BOTH Disallow lists (the * group and the AI group)

3

Upload to your website root directory

4

Test with Google's robots.txt tester

5

Monitor server logs to verify AI crawlers are accessing your site

When to Use This

New websites setting up AI crawler access

Existing sites that have blocked AI crawlers

Sites migrating to AI-optimized configuration

Need Help Implementing?

Templates are a great starting point. For custom implementation tailored to your specific needs, we can help.

Get Expert Help

More Templates

llms.txt File Template

The llms.txt file provides direct instructions to AI systems about your site. This emerging standard helps AI understand how to represent your brand.

Organization Schema (JSON-LD)

Structured data template for your organization/company. Helps AI systems understand your brand, social profiles, and contact information.

Article Schema (JSON-LD)

Structured data template for blog posts and articles. Helps AI systems understand your content's author, date, and topic.