Skip to content
Brusnica Development

robots.txt generator for AI bots

A ready robots.txt with separate rules for training bots, AI search and user requests.

Jump to the result ↓
Scenario

Model training

Collect text for new model versions

  • GPTBotOpenAI
  • ClaudeBotAnthropic
  • Google-ExtendedGoogle
  • Applebot-ExtendedApple
  • Meta-ExternalAgentMeta
  • CCBotCommon Crawl
  • BytespiderByteDance

Cite pages in answers with a link

  • OAI-SearchBotOpenAI
  • Claude-SearchBotAnthropic
  • PerplexityBotPerplexity
  • DuckAssistBotDuckDuckGo

User requests

Open a page when a person asks

  • ChatGPT-UserOpenAI
  • Claude-UserAnthropic
  • Perplexity-UserPerplexity
  • MistralAI-UserMistral

One path per line. Also applied to allowed AI bots.

robots.txt
# robots.txt — built at brusnica.dev/en/tools/robots-txt-ai-bots

# AI bots: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# AI bots: allowed (shared rules repeated, see RFC 9309)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: DuckAssistBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
Allow: /

# All other crawlers
User-agent: *
Allow: /

Which AI bots crawl websites

AI companies run three kinds of bots. Some collect text for training: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl). Others index pages for AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. The third kind opens a page when a user asks: ChatGPT-User, Claude-User, Perplexity-User.

How to block AI training but stay in AI search

Block the training bots and leave search and user bots open: that's the preset the generator starts with. Google-Extended and Applebot-Extended are control tokens: they stop Google and Apple from training models on your site without removing it from regular search. Google-Extended doesn't affect AI Overviews, which use the regular index.

Why allowed bots have to be listed separately

Under RFC 9309, a crawler follows only the most specific group that names its user agent and ignores User-agent: *. Close /admin/ to everyone, add an “allow OAI-SearchBot” group, and /admin/ is open to that bot. The generator repeats the shared Disallow rules in the allowed bots' group.

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/

User-agent: *
Allow: /
Disallow: /admin/

What robots.txt can't do

Only well-behaved crawlers follow robots.txt, and user-initiated requests sometimes skip it. Real protection needs rules at the server or CDN. Google caches the file for up to a day, and on a 5xx response many crawlers pause crawling the site.

Common questions

No. GPTBot collects training data; ChatGPT search and page fetches are done by OAI-SearchBot and ChatGPT-User.

No. It's a control token: it only governs training of Google's models and doesn't affect Search.

What's collected stays in existing datasets, but new content won't reach future training sets.

Open your-domain/robots.txt and compare it with the file; Google Search Console's robots.txt report shows parsing errors. Server logs show which bots actually visit.

Nowhere — the file is built in your browser and nothing is sent to a server.

robots.txt generator for AI bots | Brusnica Dev